BattlEye Shellcode: Part 1 Overlay Detections

Using shellcode dumps to understand how BattlEye anti-cheat detects overlay windows.

Introduction

BattlEye is a popular anti-cheat provider which has existed for ages. They protect many popular multiplayer games such as Arma 3, Rainbow Six: Siege, and Escape From Tarkov. It has always been a mystery to me about how exactly BattlEye deals with overlay detection. Despite being a lackluster anti-cheat it does do this aspect of cheat detection very well.

After obtaining a sample of a streamed BattlEye module that contains such detections, I decided to do some reversing myself to find out more. This article will fully describe window checks, information that is sent to the BattlEye server, and funny memes this Gold Standard has created.

By the end of the journey, I was able to create a tool that emulates BattlEye window detections allowing me to view an example of what the report packet may appear to look like and a method to test my in-game overlays against.

image

This also allowed me to take see what the packet might look like.

image

First Look

When loading a dump such as this into IDA, the program has no clue how to interpret the assembly therefore we need to create a function to get some useful decompiled code. After doing this, you end up getting a very large function containing about 5 thousand arguments. This makes it slightly harder to look at the disassembly; however, it is no issue. This is simply a result of the compiled code copying strings at different RSP offsets. Therefore, it interprets these offsets as different arguments.

5663 entry arguments as displayed by IDA.

After importing some ntdll.dll exports, the module references user32.dll which is where everything gets interesting.

seg000:0000000000000EE0                 mov     [rsp+SUser32DLL], 55h ; 'U'
seg000:0000000000000EE8                 mov     [rsp+arg_169], 53h ; 'S'
seg000:0000000000000EF0                 mov     [rsp+arg_16A], 45h ; 'E'
seg000:0000000000000EF8                 mov     [rsp+arg_16B], 52h ; 'R'
seg000:0000000000000F00                 mov     [rsp+arg_16C], 33h ; '3'
seg000:0000000000000F08                 mov     [rsp+arg_16D], 32h ; '2'
seg000:0000000000000F10                 mov     [rsp+arg_16E], 2Eh ; '.'
seg000:0000000000000F18                 mov     [rsp+arg_16F], 64h ; 'd'
seg000:0000000000000F20                 mov     [rsp+arg_170], 6Ch ; 'l'
seg000:0000000000000F28                 mov     [rsp+arg_171], 6Ch ; 'l'

The module begins referencing user32.dll exports such as GetTopWindow and GetWindow.

seg000:0000000000000F38                 mov     [rsp+SGetTopWindow], 47h ; 'G'
seg000:0000000000000F40                 mov     [rsp+arg_639], 65h ; 'e'
seg000:0000000000000F48                 mov     [rsp+arg_63A], 74h ; 't'
seg000:0000000000000F50                 mov     [rsp+arg_63B], 54h ; 'T'
seg000:0000000000000F58                 mov     [rsp+arg_63C], 6Fh ; 'o'
seg000:0000000000000F60                 mov     [rsp+arg_63D], 70h ; 'p'
seg000:0000000000000F68                 mov     [rsp+arg_63E], 57h ; 'W'
seg000:0000000000000F70                 mov     [rsp+arg_63F], 69h ; 'i'
seg000:0000000000000F78                 mov     [rsp+arg_640], 6Eh ; 'n'
seg000:0000000000000F80                 mov     [rsp+arg_641], 64h ; 'd'
seg000:0000000000000F88                 mov     [rsp+arg_642], 6Fh ; 'o'
seg000:0000000000000F90                 mov     [rsp+arg_643], 77h ; 'w'

After moving this string into rsp, BattlEye calls a function with the user32.dll argument and uses the return value alongside the GetTopWindow string to call another unknown function. Weird.

seg000:0000000000000FA0                 lea     rcx, [rsp+SUser32DLL]
seg000:0000000000000FA8                 call    [rsp+arg_B0E8]
seg000:0000000000000FAF                 lea     rdx, [rsp+SGetTopWindow]
seg000:0000000000000FB7                 mov     rcx, rax
seg000:0000000000000FBA                 call    [rsp+arg_B0F0]

This code does roughly this:

char* SUser32DLL = "USER32.dll";
char* SGetTopWindow = "GetTopWindow";
User32DLLBase = FunctionA(SUser32DLL);
ReturnC = (unsigned __int8 *)FunctionB(User32DLLBase, SGetTopWindow);

This should give the biggest hint into understanding what the module is doing with so many function names. It’s simply resolving the imports!

char* SUser32DLL = "USER32.dll";
char* SGetTopWindow = "GetTopWindow";
User32DLLBase = GetModuleHandle(SUser32DLL);
FGetTopWindow = (unsigned __int8 *)GetProcAddress(User32DLLBase, SGetTopWindow); // GetTopWindow Export!

This means the unknown functions are just GetModuleHandle & GetProcAddress which is used to get the address of the exported function!

Understanding WinAPI Usage

Cross-referencing the usages of these functions leads to a gold mine of data about how BattlEye handles window detections. BattlEye begins by getting the topmost window handle by its Z-order after which it iterates through the rest of the Windows.

String Comparison

The begin with the most pathetic attempt at detecting large Pay 2 Cheat providers, hard coding window names.

hCurWindow = (GetTopWindow)(0i64);
if ( hCurWindow )
{
    hTopMostWindow = 0i64;
    while ( true )                             // Loop through all windows
    {
        *CurrentWindowTitleABuffer = 0;
        CurrentWindowTextALength = -1;
        GetWindowThreadProcessId(hCurWindow, &WindowProcessId);
        if ( WindowProcessId != GetCurrentProcessId() )// Check that we are not looking at our own process
        {
            CurrentWindowTextALength = GetWindowTextA(hCurWindow, &CurrentWindowTitleABuffer[2], 128i64);
            for ( i = 0; ; ++i )
            {
                if ( i >= CurrentWindowTextALength - 5 )
                    goto LABEL_25;
                if ( *&CurrentWindowTitleABuffer[i + 2] == 'dohC' && *(v1443 + i) == 's\''
                    || *&CurrentWindowTitleABuffer[i + 2] == 'ataS' && *(v1443 + i) == '5n'
                    || *&CurrentWindowTitleABuffer[i + 2] == 'nrek' && *(v1443 + i) == 'hcle' )
                {
                    break;
                }
            }
            ...
        }
        //Further checks

Bastian did not give much effort into trying to detect Chod’s, Satan5, nor KernelCheats.

Window Size Checks

To continue, BattlEye requests the value of the window long which contains flags about the window. They check if the window is currently visible.

LABEL_25:
WindowLongStyle = GetWindowLongA(hCurWindow, GWL_STYLE);
if ( (WindowLongStyle & WS_VISIBLE) == 0 )
    goto GET_NEXT_WINDOW;

BattlEye queries the window long value again and checks the window boundaries compared to the ones of its window. GetClientRect returns a rectangle representing the window boundaries.

Pseudo Code:

//Before getting top window
strcpy(SUnityWndClass, "UnityWndClass");
hUnityWndClassWindow = FindWindowExA(0i64, 0i64, SUnityWndClass, 0i64);
GetClientRect(hUnityWndClassWindow, &ClientRectangle);
ClientToScreen(hUnityWndClassWindow, &ClientRectangle);// TOP LEFT
ClientToScreen(hUnityWndClassWindow, (RECT *)&ClientRectangle.right);// RIGHT BOTTOM

//Where we currently are
WindowLongExStyle = GetWindowLongA(hCurWindow, GWL_EXSTYLE);
GetWindowRect(hCurWindow, &WindowRectangle);
if ( GetWindowDisplayAffinity
    && GetWindowDisplayAffinity(hCurWindow, &DisplayAffinity)
    && DisplayAffinity
    && WindowRectangle.left <= ClientRectangle.left
    && WindowRectangle.top <= ClientRectangle.top
    && WindowRectangle.right >= ClientRectangle.right
    && WindowRectangle.bottom >= ClientRectangle.bottom )
{
    WindowLongStyle |= WS_CHILD;
    goto REPORT;
}

For example:

GetClientRect = (0,0) - (1277,989) // 1277x989
ClientToScreen = (323,38) - (1600,1027) // 1277x989
Essentially, ClientRect returns the dimensions meanwhile ClientToScreen translates these values to ones relative to the screen.

Hint hint, this means you can make your overlay a pixel bigger by making the game windowed to avoid this check

Window String Checks

All string comparisons appear to be backward, but this is not intentional. The strings are in little-endian format, and when the queried window title values are compared to the little-endian strings, a uint comparison occurs. Luckily, this is not an issue and everything is still readable.

  1. Notepad
CurWindowTextWLength = GetClassNameW(hCurWindow, CurWindowTitleWBuffer, 64i64);
CurrentWindowTextWLength_2 = WideCharToMultiByte(65001i64, 0i64, CurWindowTitleWBuffer, CurWindowTextWLength, ReportPacketBuffer + UnkBufferIndex + 1, 255, 0i64, 0i64);
*(ReportPacketBuffer + UnkBufferIndex) = CurrentWindowTextWLength_2;
if ( CurrentWindowTextWLength_2
    && *(ReportPacketBuffer + UnkBufferIndex) == 7
    && *(ReportPacketBuffer + UnkBufferIndex + 1) == 'etoN'
    && *(ReportPacketBuffer + UnkBufferIndex + 5) == 'ap'
    && *(ReportPacketBuffer + UnkBufferIndex + 7) == 'd' )
{
    UnusedValue = 1;
}

Every pasted P2C will be running from a hollowed notepad of course, so BattlEye of checks this. The thing is, nothing happens with the variable. Meaning I am either misinterpreting the meaning of this variable or I don’t have the rest of the shellcode for this module. BattlEye developers could have also forgotten to implement something.

  1. UnknownCheats Rainbow Six: Siege
for ( CurChildWindow = GetWindow(hCurWindow, GW_CHILD); CurChildWindow; CurChildWindow = GetWindow(CurChildWindow, GW_HWNDNEXT) )
{
    if ( GetWindowTextA(CurChildWindow, &CurChildWindowTextA, ' ')
        && ((CurChildWindowTextA == 'oceR' || CurChildWindowTextA == 'ocer') && v135 == 'li'
        || (CurChildWindowTextA == 'R-oN' || CurChildWindowTextA == 'r-oN') && v135 == 'ioce'
        || (CurChildWindowTextA == 'girT' || CurChildWindowTextA == 'girt') && v135 == 'breg'
        || CurChildWindowTextA == 'ipaR' && (v135 == 'riFd' || v135 == 'rifd' || v135 == 'iF d' || v135 == 'if d')
        || CurChildWindowTextA == 'kard' && v135 == 'aino') )
    {
        ++SusOMeter;
    }
    for ( j = GetWindow(CurChildWindow, GW_CHILD); j; j = GetWindow(j, GW_HWNDNEXT) )// Check children of parent window
    {
        if ( GetWindowTextA(j, &CurChildWindowTextA, 32i64)
            && ((CurChildWindowTextA == 'oceR' || CurChildWindowTextA == 'ocer') && v135 == 'li'
            || (CurChildWindowTextA == 'R-oN' || CurChildWindowTextA == 'r-oN') && v135 == 'ioce'
            || (CurChildWindowTextA == 'girT' || CurChildWindowTextA == 'girt') && v135 == 'breg'
            || CurChildWindowTextA == 'ipaR' && (v135 == 'riFd' || v135 == 'rifd' || v135 == 'iF d' || v135 == 'if d')
            || CurChildWindowTextA == 'kard' && v135 == 'aino') )
        {
            ++SusOMeter;
        }
        for ( k = GetWindow(j, GW_CHILD); k; k = GetWindow(k, GW_HWNDNEXT) )// Check children of child window LMFAO
        {
            if ( GetWindowTextA(k, &CurChildWindowTextA, 32i64)
                && ((CurChildWindowTextA == 'oceR' || CurChildWindowTextA == 'ocer') && v135 == 'li'
                || (CurChildWindowTextA == 'R-oN' || CurChildWindowTextA == 'r-oN') && v135 == 'ioce'
                || (CurChildWindowTextA == 'girT' || CurChildWindowTextA == 'girt') && v135 == 'breg'
                || CurChildWindowTextA == 'ipaR' && (v135 == 'riFd' || v135 == 'rifd' || v135 == 'iF d' || v135 == 'if d')
                || CurChildWindowTextA == 'kard' && v135 == 'aino') )
            {
                ++SusOMeter;
            }
        }
    }
}
if ( SusOMeter )
    break;

This is an atrocious misuse of the WinAPI. Essentially, BattlEye goes to the main window and checks the children of that window. Then it goes on to check the children of that window. That’s all. Meaning if you were to have a Parent->Child->Child->Child window, they would never check it for its title.

Blacklisted Strings:

  • Recoil
  • No-Recoi
  • No-recoi
  • Triggerb
  • RapidFir
  • Rapidfir
  • Rapid Fir
  • Rapid fi
  • drakonia

The funniest one of all is that drakonia is a moderator on UnknownCheats and all of these string references heavily relate to a popular Rainbow Six: Siege thread. Someone at BE must have been having a hard day.

At this point, if the SusOMeter value is greater than 0 (contains an illegal string), it will break out of the while loop for window checks resulting in a report packet.

WindowLongStyle |= WS_CHILD;
goto REPORT;
  1. MainWind
if ( (CombinedWindowLongFlags == 0x16CF0100 || CombinedWindowLongFlags == 0x36CF0100)
    && (*&CurrentWindowTitleABuffer[2] == 'niaM' && v1443[0] == 'dniW' || (WindowLongExStyle & 0x80000) != 0) )
{
    goto REPORT;
}
  1. BattlEye
(
*(ReportPacketBuffer + UnkBufferIndex + 1) == 'MI' && *(ReportPacketBuffer + UnkBufferIndex + 3) == 'E'
|| *(ReportPacketBuffer + UnkBufferIndex + 1) == 'TCSM'
|| *&CurrentWindowTitleABuffer[2] == 'ttaB' && v1443[0] == 'eyEl'
|| *(ReportPacketBuffer + UnkBufferIndex + 1) == 'kroW' && *(ReportPacketBuffer + UnkBufferIndex + 6) == 'Wr' && (CombinedWindowLongFlags & 0xF) != 0
|| WindowRectangle.left == -1 && WindowRectangle.top == -1
)

Inside of a comprehensive collection of window flags that are individually compared before reporting the window, Battleye checks that either of these conditions is met. In this, they check if the window name contains BattlEye or the report packet has another interesting string.

Window Long Flags

The GWL_STYLE and GWL_EX_STYLE Window Long values are combined using the OR instruction. After this BattlEye begins checking the window for hardcoded flags they have created and combined.

The following combined long flags will result in a report packet along with many more:
0x14CF0100, 0x34CF0100, 0x14EF0310, 0x34EF0310, 0x14EF0110, 0x34EF0110, 0x17090020, 0x17090000, 0x16090020, 0x94080020, 0x94080080, 0x9C080080, 0x16CF0100, 0x36CF0100

You can translate back the values to something like this:

0x9C080080 = 9c080080 | 9c080000;

GWL_EXSTYLE:
 - WS_OVERLAPPED
 - WS_POPUP
 - WS_VISIBLE
 - WS_DISABLED
 - WS_CLIPSIBLINGS
 - WS_SYSMENU
 - WS_EX_TOOLWINDOW
 - WS_EX_LEFT
 - WS_EX_LTRREADING
 - WS_EX_RIGHTSCROLLBAR
 - WS_EX_LAYERED
 - WS_EX_NOACTIVATE

GWL_STYLE:
 - WS_OVERLAPPED
 - WS_POPUP
 - WS_VISIBLE
 - WS_DISABLED
 - WS_CLIPSIBLINGS
 - WS_SYSMENU

Wrapping Up

Finally, in this first round of window checks, if you have not been caught, BattlEye will set the next window to be the child of the current window.

    hTopMostWindow = 0i64; // Code from very beginning of article.
    while ( true )
    {
        //Checks
        ...

        //Continuing enumeration
GET_NEXT_WINDOW_OR_CHILD:
        if ( !hTopMostWindow && WindowProcessId == (unsigned int)GetCurrentProcessId() && (v1124 = (void *)GetWindow(hCurWindow, GW_CHILD)) == TRUE )
        {
            hTopMostWindow = hCurWindow;
            hCurWindow = v1124;
        }

If this condition is not met it will get the next window and continue iteration as long as the report packet is not full. If the next window contains an invalid handle or the packet has exceeded its limit, BattlEye will go on to the next routine.

        else
        {
GET_NEXT_WINDOW:
            while ( true )
            {
                hCurWindow = GetWindow(hCurWindow, GW_HWNDNEXT);
                if ( hCurWindow )
                {
                    if ( BufferAllocIndex <= 0x4E7C )
                        break;
                }
                if ( !hTopMostWindow )
                    goto NEXT_CHECK;
                hCurWindow = hTopMostWindow;
                hTopMostWindow = 0i64;
            }
        }

BattlEye Report Packet

Conditions

As previously mentioned, if your window is inside the window of the game, you get reported to the BattlEye servers. BattlEye queries the file attributes if this code segment is jumped to or the following conditions are met:

// This means this logic will only execute if
// (hTopMostWindow == FALSE && NOT WS_EX_LAYERED) or
// (hTopMostWindow == TRUE && WS_EX_LAYERED)
if ( hTopMostWindow && (WindowLongExStyle & WS_EX_LAYERED) == TRUE )

Same goes with the check after:

if ( (WindowLongExStyle & WS_EX_LAYERED) != 0 && (WindowLongExStyle & WS_EX_TOPMOST) != 0 )
    goto REPORT;

Once the routine is completed, a report packet will be sent to BattlEye containing all the collected data related to detections.

Example of what a generated report packet for windows checks may appear to look like after emulating the code.

Collection

A handle to the process is opened and the image is queried.

REPORT:
pUnkBufferIndex = UnkBufferIndex + *(ReportPacketBuffer + UnkBufferIndex) + 1;
CurrentProcessHandle = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, 0i64, WindowProcessId);
CurWindowTextWLength = 128;
ProcQueryResult = CurrentProcessHandle
                && QueryFullProcessImageNameW(CurrentProcessHandle, 0i64, CurrentProcessExeName, &CurWindowTextWLength)
                && (CurWindowTextWLength = WideCharToMultiByte(0xFDE9i64, 0i64, CurrentProcessExeName, CurWindowTextWLength, ReportPacketBuffer + pUnkBufferIndex + 1, 255, 0i64, 0i64)) != 0;

If the query is successful, BE continues to also get the file size and sets it to 0 if GetFileAttributesExW returns FALSE

if ( GetFileAttributesExW(CurrentProcessExeName, 0i64, &FileAttributeInformation) )// GetFileExInfoStandard
    nFileSizeLow = FileAttributeInformation.nFileSizeLow;

The data is then packed into the 5000 byte buffer to be sent off to the servers

*(ReportPacketBuffer + pUnkBufferIndex) = BYTE4(CurWindowTextWLength_2);
v44 = pUnkBufferIndex + BYTE4(CurWindowTextWLength_2) + 1;
*(ReportPacketBuffer + v44) = CurProcessFileSizeLow;
*(ReportPacketBuffer + v44 + 4) = WindowLongStyle;
*(ReportPacketBuffer + v44 + 8) = WindowLongExStyle;
qmemcpy((ReportPacketBuffer + v44 + 12), &WindowRectangle, 0x10ui64);
BufferAllocIndex = v44 + 28;

Then it goes on to scan the next window

goto GET_NEXT_WINDOW_OR_CHILD;

Conclusion

To summarize, if your window gets detected, BattlEye will be receiving the following:

  • Executable File Size
  • Window Long Values
  • Window Rectangle
  • Window Text Length

Summary

BattlEye is no longer an anti-cheat gold standard. It has many obvious mistakes such as misuse of enumeration windows using improper Windows APIs such as GetWindow. For the most part, it relies on hard-coded window flags and string detection. More sections of the shellcode will be explored in a future article.