WIP: Run all bugreport modules against bugreport.zip in AndroidQF extraction

[DonnchaC]

We have a lot of duplication currently with separate modules for check-androidqf, check-bugreport and check-adb.

This is starting to become difficult to maintain. This PR is a POC for running all bugreport modules automatically when running check-androidqf against a AndroidQF extraction containing a bugreport.zip.

Quite a few open questions here:

• How to we keep a clean interface for loading modules rather than ad-hoc code to handle loading extra modules
• This code needs to be tested against archived/zipped AQF extractions, and folders containing the extracted filles.
• How should we namespace modules to avoid name/output file collisions.
  - [ ] How can we run tests to flag name collisions or module collobering.
• We should extend this PR to also load and check backup.ab files from an AndroidQF file by loading the check-backup modules.
• Remove duplicate modules from check-androidqf.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/mvt/android
cli.py	157	52	67%	61, 69, 88–120, 156–180, 218–219, 280–281, 288, 344–345, 375–382, 390–391
cmd_check_adb.py	10	3	70%	26–37
cmd_check_androidqf.py	57	16	72%	64–67, 75–76, 81–91, 98–99, 102
cmd_check_backup.py	63	18	71%	60, 69–70, 78–79, 85–88, 95–108, 112
cmd_check_bugreport.py	39	7	82%	51, 54–57, 70, 76
cmd_download_apks.py	86	68	21%	38–42, 51–53, 65–99, 105–112, 116–172, 175–177, 180–184
utils.py	16	5	69%	11–20
src/mvt/android/artifacts
dumpsys_adb.py	69	6	91%	45–46, 49–50, 116–117
dumpsys_appops.py	106	10	91%	27, 69–71, 117–118, 155, 171–173, 179
dumpsys_battery_daily.py	42	2	95%	42–43
dumpsys_battery_history.py	48	7	85%	47–55, 68
dumpsys_dbinfo.py	42	5	88%	60–65
dumpsys_package_activities.py	40	4	90%	12, 64, 70–71
dumpsys_packages.py	121	5	96%	18–23, 160, 167
dumpsys_platform_compat.py	22	1	95%	31
dumpsys_receivers.py	57	6	89%	24, 29, 34, 97, 103–104
file_timestamps.py	18	14	22%	12–43
getprop.py	36	4	89%	40, 53, 59, 62
processes.py	34	9	74%	20, 24–25, 31, 55, 59, 63–65
tombstone_crashes.py	150	23	85%	79, 91, 94–118, 184, 204, 209, 214, 272
src/mvt/android/modules/adb
base.py	147	111	24%	51–61, 66–73, 77–138, 142, 146–148, 157, 166–167, 171–172, 185, 198–200, 218–224, 234–269, 282–306, 309–351, 355
chrome_history.py	37	24	35%	30–38, 41, 50–56, 64–98, 101–110
dumpsys_accessibility.py	15	8	47%	26, 36–47
dumpsys_activities.py	14	7	50%	28–37, 40–45
dumpsys_adbstate.py	14	7	50%	26, 36–42
dumpsys_appops.py	14	6	57%	28, 38–44
dumpsys_battery_daily.py	13	6	54%	26, 36–42
dumpsys_battery_history.py	13	6	54%	26, 36–42
dumpsys_dbinfo.py	14	6	57%	28, 38–44
dumpsys_full.py	16	9	44%	25, 35–45
dumpsys_receivers.py	14	7	50%	26–35, 38–44
files.py	72	57	21%	37–45, 48–56, 59–70, 73–88, 93–121, 124–155
getprop.py	14	7	50%	26–35, 38–43
logcat.py	21	14	33%	25, 35–57
packages.py	168	148	12%	39–47, 50–72, 75–108, 112–161, 165–175, 180–211, 214–311
processes.py	13	6	54%	26, 36–42
root_binaries.py	25	18	28%	24, 34–36, 39–70
selinux_status.py	17	10	41%	26–35, 38–48
settings.py	25	18	28%	26–35, 38–58
sms.py	79	60	24%	56–65, 68–69, 77–90, 98–127, 137–151, 154–179
whatsapp.py	53	39	26%	31, 41–42, 50–60, 68–103, 106–113
src/mvt/android/modules/androidqf
base.py	43	2	95%	73–74
dumpsys_packages.py	24	3	88%	52–53, 56
files.py	78	21	73%	73, 82–84, 88, 92–101, 106–109, 115, 124–129
logfile_timestamps.py	24	24	0%	6–62
packages.py	57	6	89%	46–51, 79, 84, 104
settings.py	24	2	92%	51–52
sms.py	49	13	73%	48, 51, 56–57, 73–78, 81, 85–91, 96–97
src/mvt/android/modules/backup
base.py	33	2	94%	64–65
helpers.py	22	1	95%	27
sms.py	33	4	88%	40, 44, 47–48
src/mvt/android/modules/bugreport
accessibility.py	17	1	94%	51
adb_state.py	16	1	94%	51
base.py	58	18	69%	49–50, 55–56, 63–68, 72, 87–88, 94, 97–102
fs_timestamps.py	16	2	88%	44–45
getprop.py	26	5	81%	51–52, 57–60
packages.py	24	1	96%	56
tombstones.py	24	11	54%	45–61
src/mvt/android/parsers
backup.py	109	9	92%	62, 102–103, 109, 129, 132, 175, 190–191
src/mvt/common
artifact.py	10	1	90%	22
cmd_check_iocs.py	38	29	24%	26–36, 39–80
command.py	136	50	63%	68–74, 80–98, 106–119, 129–157, 163–167, 170–172, 178, 212, 229, 233–234
indicators.py	317	65	79%	39–41, 54–60, 139, 151, 157, 163, 169, 191, 205–210, 293, 305, 317, 359–362, 379, 404–422, 449, 471, 502, 521, 536–542, 555–563, 574, 598, 623, 632–639, 654, 679–692, 703, 706, 733, 780, 798
logo.py	40	33	18%	16–74, 78–85
module.py	122	39	68%	71–75, 80–84, 99–119, 160, 169, 174, 184, 203–204, 220–221, 237–251
options.py	13	3	77%	27–33
updates.py	144	118	18%	27–34, 39–52, 57–65, 68–70, 77–85, 88–90, 93–105, 108–123, 126–164, 171–200, 207–217, 220–250
url.py	25	6	76%	327, 366, 372–376
utils.py	108	37	66%	49–51, 64, 96–97, 111, 125–126, 165–182, 197–198, 211–212, 219–228, 243, 253, 261
virustotal.py	23	13	43%	25–52
src/mvt/ios
cli.py	139	62	55%	61, 69, 98–142, 161–184, 225–226, 259–280, 303–310, 318–319
cmd_check_fs.py	13	4	69%	28–40, 43
decrypt.py	114	92	19%	33–36, 39, 48–56, 61–64, 73–123, 131–181, 192–221, 227–231, 244–255
versions.py	32	3	91%	21, 30, 48
src/mvt/ios/modules
base.py	90	23	74%	54, 64, 71–96, 114, 122, 129, 137–138, 157–160, 195–196
net_base.py	123	48	61%	68–70, 100–101, 182–237, 252–263, 275, 320–321, 329–330, 334
src/mvt/ios/modules/backup
backup_info.py	29	2	93%	43, 79
configuration_profiles.py	67	46	31%	43–48, 58, 61–88, 103–178
manifest.py	81	8	90%	59, 66, 112–118, 123, 169–170
profile_events.py	51	32	37%	44, 55, 58–67, 71–97, 103–110, 113
src/mvt/ios/modules/fs
analytics.py	65	50	23%	34, 44, 52–77, 80–137, 140–143, 146–152
analytics_ios_versions.py	36	26	28%	30, 40, 48–86
cache_files.py	46	35	24%	24, 34–45, 48–62, 65–80, 92–99
filesystem.py	42	8	81%	52, 56–57, 61, 77–78, 89–90
net_netusage.py	18	10	44%	34, 44–57
safari_favicon.py	37	26	30%	31, 41, 50–60, 63–115, 118–124
shutdownlog.py	65	54	17%	30, 40, 49–69, 72–127, 130–133
version_history.py	20	9	55%	32, 42, 50–65
webkit_base.py	22	16	27%	17–24, 27–38
webkit_indexeddb.py	13	4	69%	34, 44, 53–54
webkit_localstorage.py	12	4	67%	32, 42, 51–52
webkit_safariviewservice.py	10	3	70%	32, 42–43
src/mvt/ios/modules/mixed
applications.py	75	46	39%	50–57, 61–97, 103–111, 117–122, 132–144, 150, 152–154
calendar.py	49	2	96%	75–78
calls.py	22	10	55%	41, 53–82
chrome_favicon.py	36	23	36%	42, 50–60, 66–104
chrome_history.py	30	17	43%	44, 54–61, 67–102
contacts.py	28	17	39%	45–75
firefox_favicon.py	32	19	41%	43, 52–62, 68–106
firefox_history.py	30	17	43%	47, 55–62, 68–101
global_preferences.py	25	1	96%	45
idstatuscache.py	56	39	30%	46, 55–72, 75–105, 110–120
interactionc.py	55	40	27%	251–275, 281–320
locationd.py	83	65	22%	58–70, 73–133, 136–155, 160–172
osanalytics_addaily.py	31	17	45%	45, 56–63, 70–98
safari_browserstate.py	73	27	63%	68, 71–75, 96–98, 112–132, 167, 173–180
safari_history.py	70	44	37%	48, 59–98, 107, 110–113, 116–151, 163–171
shortcuts.py	67	50	25%	47–55, 71–78, 84–153
sms.py	69	13	81%	74, 86, 110–127, 140, 154
sms_attachments.py	42	24	43%	44, 57–74, 97–128
tcc.py	80	28	65%	68, 85, 108–125, 140–143, 163–205
webkit_resource_load_statistics.py	52	10	81%	67–68, 93–94, 128–135
webkit_session_resource_log.py	75	52	31%	56–66, 70, 73–113, 119–149, 156–173
whatsapp.py	51	38	25%	43–48, 56–63, 69–135
TOTAL	6602	2523	62%

Tests	Skipped	Failures	Errors	Time
110	1 💤	0 ❌	0 🔥	9.440s ⏱️

[DonnchaC]

Closing this as I have reworked this feature in #606