I am trying to run a crawler on CSV file located in s3 where |~| is the delimiter. I gave the delimiter as below:
I ran the crawler but it is not able to read the data and creates an empty table. I have more than 600 columns and not sure how to create a GROK pattern for this. Hive provides a way to remove multi-delimiter easily like this : 'field.delim'='|~|' I tried to do the same using Athena but no luck.I don't want to re-process the whole file again(20GB) by using python code just to replace/remove the delimiter.
Has anyone had the same issue? Any help would be appreciated.
Thanks!