Conflicting coreMQTT and coreHTTP network_transport modules (CA-348)

[rodmaz]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate.
• Provided a clear description of your suggestion.
• Included any relevant context or examples.

Issue or Suggestion Description

Both coreMQTT and coreHTTP components uses different port/network_transport modules but sharing same symbols.
When integrating both components, linker complains about duplicated symbols.
Besides, network_transport.cin coreMQTT seems more sophisticated that the one used in coreHTTP. Why?

[avsheth]

Hi @rodmaz
Ideally the build system should pick the network_transport.o object based on the order of the components in the CMakeLists and linker shouldn't have seen multiple definitions. This theory holds for the ota_http example on release/202210.01-LTS branch. But I suppose that's true if both components dependency for a single library. Can you help share the branch you're on and share the error logs as well as your CMakeLists changes?

But agree that it kind of introduces a subtle bug as well where one gets picked over the other and another one is ignored silently.

About differences between the two files, there are couple of reasons:

1. The latest LTS release introduces new APIs for MQTT transport to set the connect, receive and send timeouts individually at runtime to help applications control the latency and wait time for various requests (publish, subscribe, suback etc). Since this was more from the context of device control over MQTT, we haven't yet added the same in the HTTP transport.
2. Latest AWS IoT LTS release, 202406.01, has moved to Modular OTA now. For which, application can control and customise the transport, job document etc, to suit their needs. For reference, MQTT stream is used as streaming medium for the OTA, hence coreMQTT transport has been updated with those API to manage OTA latency as well.

Hope this helps.

[cmorganBE]

Seeing the same here with 202406.01 release, ESP-IDF v5.2.2.

Tried fixing it by adding REQUIRES entries to my projects idf_component_register(), thinking that this would cause coreMQTT to get picked up before coreHTTP.

This change fixed the compile issue with missing symbols for vTlsSetConnectTimeout() but the problem has moved to linking and there are now conflicting symbols:

components/esp-aws-iot/libraries/coreMQTT/port/network_transport/network_transport.c:123: multiple definition of `espTlsTransportSend';

The same error happens on these symbols:

xTlsConnect
xTlsDisconnect
espTlsTransportSend
espTlsTransportRecv

As they are apparently defined in coreHTTP's network_transport.c

[cmorganBE]

In my case its blocking me entirely due to this issue. I'll likely work around it by moving one of the files out of the way in the component but that's only a temporary solution.

[cmorganBE]

@rodmaz did you figure out a work around to this that doesn't involve modifying the esp-aws-iot component itself?

[rodmaz]

I created symlinks from esp-aws-iot coreHTTP and coreMQTT source code into my IDF project components and modified the CMakeLists.txt, which are mine now. Not the optimal solution but it works just fine.
Also network_transport is now mine. Ported into my own networking component.
Hope it helps!

[cmorganBE]

@rodmaz are you using both modules? In my case I'm only using coreMQTT.

I'm also doing this in my top level CMakeLists.txt file:

list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries)

which I think means the esp-idf build system will auto-discover all of the nested components.

If I remove that EXTRA_COMPONENT append then I think I'd have to manually include all of the components I wanted in the esp-aws-iot/libraries/ subdirectory... not ideal but not terrible either.

[rodmaz]

I don't have that. I use both modules but I use symlinks to bring each component from ESP AWS IoT I need into my project.
So my project components folder has a folder for coreMQTT and another folder for coreHTTP etc.
e.g. whenever a component in my project uses coreMQTT, this component itself will declare its dependance on it in CMakeLists.txt:

idf_component_register(
  SRCS
    "my_http_client.c"
  INCLUDE_DIRS
    "include"
  PRIV_REQUIRES
    backoffAlgorithm
    posix
    my_common
    my_logging
    my_networking
  REQUIRES
    coreHTTP
    esp_wifi
)

[cmorganBE]

ahh k.

I just tested and this is a diff of what works as well as an alternate to the symlink:

+# We want to use the below, except if we do there is a conflict between coreHTTP and coreMQTT
+# related to network_transport.c, the build system finds the coreHTTP one even though coreMQTT
+# is being used
+#
+#list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries)
+#
+# So instead the components used are added manually
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/aws-iot-core-mqtt-file-streams-embedded-c)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/backoffAlgorithm)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/common)
+# Can't include this AND coreMQTT at the same time
+#list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/coreHTTP)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/coreJSON)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/coreMQTT)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/coreMQTT-Agent)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/corePKCS11)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/Device-Defender-for-AWS-IoT-embedded-sdk)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/Device-Shadow-for-AWS-IoT-embedded-sdk)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/Fleet-Provisioning-for-AWS-IoT-embedded-sdk)
+list(APPEND EXTRA_COMPONENT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/components/esp-aws-iot/libraries/Jobs-for-AWS-IoT-embedded-sdk)