Cemus Report Verification

Verify the Location of Cemus Config Files

To verify the location of the Daily Cemus, Monthly Cemus, GRT Cemus reports log into the management node and run the following command:

hdfs dfs -ls /data/streaming/
  • For DailyCemusReport, check the following files should be available at the location /data/streaming
    • DailyCemusReport.properties

    • DailyQueriesCemus.txt

  • For MonthlyCemusReport, check the following files should be available at the location /data/streaming
    • MonthlyCemusReport.properties

    • MonthlyQueriesCemus.txt

  • For GRTCemusReport, check the following files should be available at the location /data/streaming
    • GRTCemusReport.properties
    • GRTQueriesCemus.txt
    • grt-protocol-values.csv
  • protocol-values.txt

This file consists of the list of protocols for which daily, monthly and GRT reports will be generated. This file remains common for daily, monthly and GRT aggregations and should also be available at the location /data/streaming. To verify its location, run the below command:

hdfs dfs -ls /data/streaming/protocol-values.txt

Output:

-rw-r--r--   3 root hadoop      75428 2020-03-31 16:37 /data/streaming/protocol-values.txt

Verify the Content of Cemus Config Files:

The content of the Daily Cemus, Monthly Cemus, GRT Cemus reports can be verified by the following steps:

  1. For DailyCemusReport

    • To verify the content of DailyCemusReport.properties file, run the following command:
      hdfs dfs -cat /data/streaming/DailyCemusReport.properties

      Output:

      dbName=kafkaconnectdb
      input.tableName=5min_points_new
      reportType=daily
      query.file=/data/streaming/DailyQueriesCemus.txt
      download.threshold=1000
      upload.threshold=1000
      locality.code=UK
      notAvailableString=NotAvailable
      unknownString=Unknown
      cemusReportRootDir=/data/mrx/customer/cemus-report
      single.file.report=true
      protocol.file=/data/streaming/protocol-values.txt
      sftp.location=/root/sftp/cemus

      Notes:

      1. Here, download.threshold and upload.threshold represents the threshold value in bytes for downloading and uploading of the files. Both the property values can be customized based on the client requirement.
      2. sftp.location represents the location where all the reports can be placed.
      3. The value for single.file.reportshould be set to true.
  2. For MonthlyCemusReport

    • To verify the content of MonthlyCemusReport.properties, run the following command:
      hdfs dfs -cat /data/streaming/MonthlyCemusReport.properties

      Output:

      dbName=kafkaconnectdb
      input.tableName=daily_points_new
      input.tableName2=montly_points_new
      reportType=monthly
      query.file=/data/streaming/MonthlyQueriesCemus.txt
      download.threshold=3000
      upload.threshold=3000
      locality.code=UK
      notAvailableString=NotAvailable
      unknownString=Unknown
      cemusReportRootDir=/data/mrx/customer/cemus-report
      single.file.report=true
      protocol.file=/data/streaming/protocol-values.txt
      sftp.location=/root/sftp/cemus
  3. For GRTCemusReport

    • To verify the content of GRTCemusReport.properties, run the following command:

      hdfs dfs -cat /data/streaming/GRTCemusReport.properties

      Output:

      dbName=kafkaconnectdb
      input.tableName=hourly_points_new
      reportType=GRT
      query.file=/data/streaming/GRTQueriesCemus.txt
      download.threshold=3000
      upload.threshold=3000
      locality.code=UK
      notAvailableString=NotAvailable
      unknownString=Unknown
      cemusReportRootDir=/data/mrx/customer/cemus-report
      single.file.report=true
      protocol.file=/data/streaming/protocol-values.txt
      grt.protocol.file=/data/streaming/grt-protocol-values.txt
      sftp.location=/root/sftp/cemus/GRT
      smtp.server.ip=192.168.104.25
      smtp.port=25
      sender.address=support-mrx@guavus.com
      receiver.addresses=sample1@host.com,sample2@host.com

      Notes:

      1. Here,download.threshold and upload.threshold represents the threshold value in bytes for downloading and uploading of the files. Both the property values can be customized based on the client requirement.
      2. sftp.location represents the location where all the reports can be placed.
      3. The value for single.file.report should be set to true.
    • grt-protocol-value.csv

      • To verify the content of grt-protocol-value.csv, run the following command:

        hdfs dfs -cat /data/streaming/grt-protocol-values.csv

        Output:

        protocol,audio_coded_kbps,video_coded_kbps
        whatsapp,25,240
        facebook,30,500
        skype,50,390
        viber,45,980
        facetime,32,1000
        googleduo,60,590
        googlehangout,55,450
        other voip,35,335
        

        This file is used specifically for report Mobile-UK-2-PGW_04.16.19 08.00 AM.csv report generation. The property values like protocol, audio codec and video codec can be changed by the customer accordingly or customer can also add a new protocol or remove existing protocol as per the requirements.

        protocol-values.txt

      • To verify the content of protocol-values.txt file, run the following command.

        hdfs dfs -cat /data/streaming/protocol-values.txt|head

        The sample output may resemble as follows:

        youtube=YouTube
        https=HTTPS
        google-play=Google Play
        whatsapp-voice=WhatsApp Voice
        whatsapp-transfer=WhatsApp Transfer
        wechat-unclassified=Wechat
        snapchat-unclassified=Snapchat
        skype-streaming-video=Skype Streaming-Video
        skype-unclassified=Skype
        whatsapp-unclassified=Whatsapp

      Verify the DailyCemusReport

      This is the output file generated based on the configurations in DailyCemusReport.properties and DailyQueriesCemus.txt file. To verify the location of the file, run the following command:

      hdfs dfs -ls /data/mrx/customer/cemus-report/daily

      For example,

      hdfs dfs -ls /data/mrx/customer/cemus-report/daily/20210712

      The sample output may resemble as follows:

      Found 10 items:
      rw-rr-   3 root hadoop     758682 2021-07-12 08:09 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServicesCorporate_ALL_ALL_D_2021-07-12-08-03-56.csv
      rw-rr-   3 root hadoop    1539476 2021-07-12 07:58 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServicesCorporate_ALL_TECH_D_2021-07-12-07-47-47.csv
      rw-rr-   3 root hadoop    3961523 2021-07-12 08:03 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServicesCorporate_DS_ALL_D_2021-07-12-07-58-08.csv
      rw-rr-   3 root hadoop    6010909 2021-07-12 07:47 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServicesCorporate_DS_TECH_D_2021-07-12-07-37-17.csv
      rw-rr-   3 root hadoop     751497 2021-07-12 07:37 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServices_ALL_ALL_D_2021-07-12-07-31-55.csv
      rw-rr-   3 root hadoop    1916603 2021-07-12 07:26 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServices_ALL_TECH_D_2021-07-12-07-15-55.csv
      rw-rr-   3 root hadoop    3920690 2021-07-12 07:31 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServices_DS_ALL_D_2021-07-12-07-26-20.csv
      rw-rr-   3 root hadoop    5947961 2021-07-12 07:15 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServices_DS_TECH_D_2021-07-12-06-56-35.csv
      rw-rr-   3 root hadoop        644 2021-07-12 08:18 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServices_Time_BW_Distribution_ALL_D_2021-07-12-08-15-39.csv
      rw-rr-   3 root hadoop       1802 2021-07-12 08:15 /data/mrx/customer/cemus-report/daily/20210712/Umbrella_UK_DataServices_Time_BW_Distribution_TECH_D_2021-07-12-08-09-38.csv

      Verify the MonthlyCemusReport

      This is the output file generated based on the configurations in MonthlyCemusReport.properties and MonthlyQueriesCemus.txt file. To verify the location of the file, run the following command:

      hdfs dfs -ls /data/mrx/customer/cemus-report/monthly

      For example:

      hdfs dfs -ls /data/mrx/customer/cemus-report/monthly/20210710

      The sample output may resemble as follows:

      Found 4 items
      rw-rr-  3 root hadoop  417 2021-07-10 03:52 /data/mrx/customer/cemus-report/monthly/20210710/Umbrella_UK_DataServices_BW_Distribution_ALL_M_2021-07-10-03-51-22.csv
      rw-rr-  3 root hadoop  711 2021-07-10 03:51 /data/mrx/customer/cemus-report/monthly/20210710/Umbrella_UK_DataServices_BW_Distribution_TECH_M_2021-07-10-03-48-00.csv
      rw-rr-  3 root hadoop  359 2021-07-10 03:53 /data/mrx/customer/cemus-report/monthly/20210710/Umbrella_UK_DataServices_BW_Percentile_ALL_M_2021-07-10-03-53-13.csv
      rw-rr-  3 root hadoop  1004 2021-07-10 03:53 /data/mrx/customer/cemus-report/monthly/20210710/Umbrella_UK_DataServices_BW_Percentile_TECH_M_2021-07-10-03-52-48.csv

      Verify the GRTCemusReport

      This is the output file generated based on the configurations in GRTCemusReport.properties and GRTQueriesCemus.txt file. To verify the location of the file, run the following command:

      hdfs dfs -ls /data/mrx/customer/cemus-report/GRT

      For example:

      hdfs dfs -ls /data/mrx/customer/cemus-report/GRT/20210711

      Output:

      Found 2 items
      rw-rr-   3 root hadoop       1528 2021-07-11 06:48 /data/mrx/customer/cemus-report/GRT/20210711/Mobile-[UK]-1-PGW_07.11.21_06.48_AM.csv
      rw-rr-   3 root hadoop       8484 2021-07-11 06:49 /data/mrx/customer/cemus-report/GRT/20210711/Mobile-[UK]-2-PGW_07.11.21_06.48_AM.csv