We’ve all heard of O365 (Office 365) or M365 (Microsoft 365) (depending on the name of this product this week), but how many of us responders know how to investigate a Google Workspace incident? If you’re an MSSP, vendor, or even working your own Incident Response (IR), you may find yourself running into more and more Google Workspace environments. Whether it’s investigating a suspected malicious email or a compromised user, do you know what logs exist? Are you familiar with the capabilities of your license? Do you know how to use the dashboard? If not, then this blog post is for you! Although I won’t be able to dive into every single feature of Google Workspace, I want to hit some of the key items to consider and review.
The Rundown:
Google Apps > Google Apps for Work > G Suite > Google Workspace
Like many solutions, this product has gone through many names
Workspace has multiple editions with different features and capabilities. These include Starter, Standard, Plus, Business, Enterprise, Education and G Suite. There are many others, as well as custom set-ups with addons and features specific to you
Visibility and logging can vary based off of your edition
The “Starter” edition has extremely limited logging capabilities
Critical logs for investigations are Admin Audit Logs, User Log Events, Admin Activities, SAML, OAuth (though there are many others)
Logs are retained for 6 months (if you need to extend this, forward these to a SIEM/Log aggregator, or Google Cloud)
Workspace uses Organizational Units and Groups to assign permissions
Logs can be accessed via the Workspace Admin Interface or the API
Note that each have their pros and cons. For example, a cap on exported results
Exported results can be limited to 10,000 or 100,000 results through the Workspace Admin (depending on the data type)
Can forward logs from Workspace to Google Cloud (log aggregation, yay!)
Not all logs are enabled to ship to Google Cloud
Depending on your license, Workspace may offer “detections” through the Security Dashboard
Investigation Tool allows users to search through various log sources
If purchased, Google Vault can provide additional features to aid investigations using “Matters” for searching, archiving and exporting
Alright, alright. That was a decent rundown! I wish I could give all the features, but there’s just too many! So, let’s focus on some of the critical ones, right? Before we get started with the analysis, lets come up with a few ideas and questions that we may need to understand.
Is this email confirmed malicious or suspicious? If so, why?
What is the intent of this email? Did it contain a malicious attachment or link? If so, have these been analyzed for additional intelligence and Indicators of Compromise (IOCs)?
Did this lead to a credential harvesting domain? If so, did the user enter their credentials?
Did this lead to the download of malware? If so, what is the current status of the host and user?
How many users received this email?
Did this come from a trusted sender, was it spoofed, or a random email address?
Has this sender been blocked?
If this is malicious, has this been removed from user’s inboxes?
For users that interacted with the email, have they been investigated? For example:
Has their account been disabled and the password reset?
Has their account been reviewed for suspicious activity? This includes email forwarding rules, inbox rules, sent emails, deleted items, newly added 2FA device, etc. Essentially, anything that occurred during the time of impact that was not authorized
Has their host been isolated and reviewed?
Have known IOCs been blocked and searched for to scope the incident? This may include
Sender email address
Message ID
Subject
Attachments
Links
IP addresses (if available)
Contents of the email body
It goes without saying that the above items mentioned are essentially at the point of where we assume compromise. Essentially, the idea is the understand the incident at hand and contain it. Especially when it comes to email compromises, you want to ensure that a domino effect doesn’t occur and users around your organization keep getting popped. For example, one user is compromised, this account is used to send another phishing email to internal employees, more users are compromised, and so on and so forth. A business email compromise loop or a lovely game of whack-a-mole!
So, let’s assume that our user has been compromised and we’re dealing with an account takeover. What this means is that an authorized actor gained access to a legitimate users account. What logs exist in Google Workspace to investigate this? Again, we want to understand the incident by scoping it, collecting intelligence, which can drive our analysis, contain the incident, and eventually eradicate the threat.
Let’s start by disabling the impacted account and resetting the password. Where do we go for this? We’ll want to access the Google Workspace Admin dashboard. Specifically, the “Directory”, which contains our users. From here, search the user in question and select “More Options” and “Suspend User”. Note that you can also click a user to access their "user" dashboard. From here, you can perform many of the below actions.
Next, let’s change that user’s password. Luckily, this is in the same “User Directory” dashboard. Again, search the user in question and now select “Reset password”. It’s critical to also reset the user’s “Sign-in cookies”, which prevents the user from signing in using an old password through session tokens. This can be done by selecting the user in question > Security > Sign-in Cookies > Reset. Its important to note that when the user is suspended, their sign-in cookies and tokens will be reset.
Excellent! Now the account is disabled! Let’s work to understand when the unauthorized access occurred. This can help give us a timeframe of the incident, which can lead to additional IOCs to drive our analysis. For this, let's take a look at the "User Log Events" within the Reporting > Audit and Investigation section. Note that this tab will provide the majority of our logs regarding unauthorized access. What are we looking for here? Well.. unauthorized access! Take a look at sign-ins at anomalous times, unexpected IP addresses, etc. Also take a look at the SAML and OAuth logs! Especially if we think a malicious third-party Workspace app was leveraged for initial access! These apps may be used to gain access to an account via OAuth! Pro Tip: Once you find a malicious IP address, use this to look for other unauthorized access! It goes without saying that if a malicious app is identified, be sure to revoke its access and remove it.
Note that the "User Log Events" will be used for the majority of analysis. However, if you're dealing with a compromised Admin account, you'll want to review the "Admin Audit Log" as well.
Next, let's investigate emails! This can be done by going into “Reporting” and selecting “Email Log Search”. From here, we can search for emails based off of a number of criteria. Once the email is identified, we can use this to scope the incident and determine how many users received the malicious email.
From here, you can collect information such as email header information, if it was delivered successfully, and if the email was read or seen.
Once we identify the email, we can work on quarantining it from user’s inboxes using “Quarantine”.
Note that ideally, you’d want to have an edition that includes the Investigation Tool, as this will make responding much more fluid and seamless. Using this tool, you can search for the malicious email and quarantine it from inboxes from a single dashboard. As mentioned, there are many other logs as well. You'd want to ensure the user is fully investigated to understand the incident and not miss unauthorized changes or activity. For example, reviewing Google Drive logs for suspicious activity, Takeout logs for exported data, and more!
Whelp, in the hope of shortening this blog post, let’s wrap this up by restoring the compromised user! Obviously, there’s many other things we should do for this investigation that I didn’t show. For example, analyzing the malicious email and determining its intent and additional IOCs. Investigating the host, reviewing network-based logs such as IDS/IPS, proxy, NetFlow, firewall, etc. Hopefully this can shed some light on responding to Workspace compromises!
Resources: