SOAR Error Message: "Ingestion error: a bytes-like object is required, not 'str' "

Applicability

Description

The SOAR app Microsoft Exchange On-Premise EWS ingests emails from a On-Premise EWS. 

Using version 3.13.2, depending on the ingested emails encoding, this error message could be seen in spawn.log

2024-06-27T18:55:53.709297 PID:17423 TID:17423 : ERROR: SPAWN3 : /srv/phantom/alchemy/connectors/spawn3/spawn3.cpp : 1139 : EWSOnPremConnector :Exception occurred. Exception: a bytes-like object is required, not 'str' 
Traceback (most recent call last): 
 File "/home/phantomnri/phantom/apps/microsoftexchangeonpremiseews_badc5252-4a82-4a6d-bc53-d1e503857124/process_email.py", line 985, in _handle_mail_object 
 self._handle_body(body, self._parsed_mail, i, email_id) 
 File "/home/phantomnri/phantom/apps/microsoftexchangeonpremiseews_badc5252-4a82-4a6d-bc53-d1e503857124/process_email.py", line 332, in _handle_body 
 file_data = file_data.replace("\r\n ", "").replace("\r\n", " ").replace("\\n", " ") 
TypeError: a bytes-like object is required, not 'str'

Investigation

This is a bug of this app in this particular version that has been filed as a known problem with code SOARHELP-3527.

The root cause is a failure to fit a byte stream that comes within an email into a particular encoding system.

The consequence of facing it is that the artifact ingestion stops at the point of the failure and other artifacts will not be ingested.

For example, having an email with 40 artifacts where this issue happens processing the 4th one, the app will create a new container in SOAR with only 3 artifacts and 37 will not be ingested.

Resolution

The solution of this issue will come in a future release of this app that, by the time of writing this KB, is not decided yet.

Workaround:

The app can be fixed changing the code of the _get_string method adding a try block and using utf-8. 

def _get_string(self, input_str, charset): 
 output_str = None 
 try: 
 if input_str: 
 output_str = UnicodeDammit(input_str).unicode_markup.encode(charset).decode(charset) 
 except Exception: 
 try: 
 charset = 'utf-8' 
 output_str = UnicodeDammit(input_str).unicode_markup.encode(charset).decode(charset) 
 except Exception: 
 self.debug_print(f"Error occurred while converting to string with specific encoding: {charset=}") 
 return output_str

Related Links

https://splunkbase.splunk.com/app/5788/