Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pi 5 UARTs broken in 6.6.44 build of rpi-6.6.y. #6365

Open
pelwell opened this issue Sep 17, 2024 · 16 comments
Open

Pi 5 UARTs broken in 6.6.44 build of rpi-6.6.y. #6365

pelwell opened this issue Sep 17, 2024 · 16 comments

Comments

@pelwell
Copy link
Contributor

pelwell commented Sep 17, 2024

Describe the bug

As detailed by @lgeitner in the thread starting at #6144 (comment), the MeteoPi application which requires UART access seems to be broken in the 6.6.44 build of rpi-6.6.y. It has also been shown to have worked in the 6.6.42 build. The corresponding source commits for those two builds are:

"kernel: Bump to 6.6.42" (1969fd7): bfbd468
"kernel: Bump to 6.6.44" (44a287f): 2806f39

Steps to reproduce the behaviour

Run MeteoPi. Other test cases have yet to be determined.

Device (s)

Raspberry Pi 5

System

Linux 6.6.44-v8-16k+ #1789 SMP PREEMPT Mon Aug 5 15:24:18 BST 2024 aarch64 GNU/Linux

Logs

N/A

Additional context

No response

@pelwell
Copy link
Contributor Author

pelwell commented Sep 17, 2024

The current list of suspect commits is:

2806f39528464 Revert "mfd: Partial revert of Add rp1 driver"
d0ea54e1c941a mfd: Partial revert of Add rp1 driver
216df57950849 DTS: bcm2712: enable SD slot CQE by default on Pi 5
3c319a466a1c7 dtoverlays: Document display_[width|height] on hd44780-lcd overlay
e94e761305fa2 dtoverlays: Add overlay for HD44780 via I2C PCF8574 backpack
a4bf61fad9fe1 fixup! pinctrl: bcm2712 pinctrl/pinconf driver
16d0ee22d2c0b hwmon: (adt7410) Add DT compatible strings
05e3687c6c973 overlays: i2c-rtc: Correct bq32000 property name
e9294823cf020 spi: dw: Clamp the minimum clock speed
199e611183de0 spi: dw: Fix non-DMA transmit-only transfers
70636ad110715 ARM: dts: bcm2712: Fix invalid polling-delay-passive setting
485d11cfa7df2 drm/vc4: Disable the 2pixel/clock odd timings workaround for interlaced
062434ab3be76 dts: rp1: restrict i2s burst lengths to 4
b6b4260fa546d sound/soc: dwc-i2s: choose FIFO thresholds based on DMA burst constraints
5112fd8cce4f1 DT: bindings: add a dma-maxburst property to snps,designware-i2s
f3cb675102a2a tty/serial: pl011: restrict RX burst FIFO threshold
1c4c90982462e Revert "spi: dw-dma: Get the last DMA scoop out of the FIFO"
ce56098eb4dc2 drivers: dw-axi-dmac: make more sensible choices about memory accesses
4c1a665b465fa dts: rp1: hobble DMA AXI burst lengths
3af7822df36e3 spi: dw: don't immediately kill DMA transfers if an error occurs
cd9084ceb606a spi: dw: Save bandwidth with the TMOD_RO feature
6014649de765a spi: dw: Save bandwidth with the TMOD_TO feature
31b9871b8895d staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1

I'll try and create a PR from the 6.6.44 build head (2806f39) but with those commits reverted.

@pelwell
Copy link
Contributor Author

pelwell commented Sep 17, 2024

#6366 is the PR that will create the trial builds. After each update to it, wait about 45 minutes for the builds to complete then run sudo rpi-update pulls/6366.

@pelwell
Copy link
Contributor Author

pelwell commented Sep 17, 2024

Do you want to reconsider any of the above patches, @P33M?

@P33M
Copy link
Contributor

P33M commented Sep 17, 2024

f3cb675102a2a tty/serial: pl011: restrict RX burst FIFO threshold may be causing side-effects. This isn't critical to UART operation with the restricted DMAC burst sizes - the RX timeout interrupt will gather any remaining bytes.

@P33M
Copy link
Contributor

P33M commented Sep 17, 2024

This might be the same bug - long TX bursts appear to drop data on Pi 5 - https://forums.raspberrypi.com/viewtopic.php?t=376532

@pelwell
Copy link
Contributor Author

pelwell commented Sep 17, 2024

With f3cb675 I see data loss at the end of transmission:

00009300: 72 69 61 6c 3a 20 6e 6f : 20 70 6c 61 74 66 6f 72
00009308: 20 44 4d 41 20 70 6c 61 : 6d 20 64 61 74 61 0a
00009310: 74 66 6f 72 6d 20 64 61 :
00009318: 74 61 0a                :

(where 931a is the last byte, and everything before 9300 matches)
With it reverted, I've not seen any data loss yet.

I've updated #6366 to be top-of-tree rpi-6.6.y but with f3cb675 reverted. As before, wait about 45 minutes then run sudo rpi-update pulls/6366.

pelwell added a commit to pelwell/linux that referenced this issue Sep 17, 2024
This reverts commit f3cb675.

Data loss has been observed on the Pi's PL011 UARTs since f3cb675 was
merged. The loss appears to be on reception, not transmission.

Revert the commit as a workaroud and potential fix.

Link: raspberrypi#6365
Link: raspberrypi#6144 (comment)
Link: https://forums.raspberrypi.com/viewtopic.php?t=376532

Signed-off-by: Phil Elwell <[email protected]>
@lgeitner
Copy link

ran sudo rpi-update pulls/6366

received the error "Invalid artifact specified. Response: 404."
please see below

pi@cumulusmxkitchen:~ $ sudo rpi-update pulls/6366
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Performing self-update
*** Relaunching after update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
FW_REV:
BOOTLOADER_REV:d05f05c94fd7a1916942e56d7f486c142ae5661e
WANT_32BIT:0 WANT_64BIT:1 WANT_PI4:1 WANT_PI5:1
*** Downloading specific artifact revision (this will take a few minutes)
curl -L https://builds.raspberrypi.com/github/linux/8aa0ad843d49a58d738177fd892ec547015dd231/bcmrpi | zcat | tar xf - -C //root/.rpi-firmware --strip-components=2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0

gzip: stdin: unexpected end of file
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
Invalid artifact specified. Response: 404.
pi@cumulusmxkitchen:~ $

@pelwell
Copy link
Contributor Author

pelwell commented Sep 17, 2024

I thought this might happen. I updated the PR to include a merge-ready commit message, and you came online and tried to update in the interval while it was being rebuilt. It should be good now.

@lgeitner
Copy link

ran sudo rpi-update pulls/6366 then sudo reboot

the meteopi is not working!

the command uname -a displays the following:

pi@cumulusmxkitchen:~ $ uname -a
Linux cumulusmxkitchen 6.6.51-v8-16k+ #1 SMP PREEMPT Tue Sep 17 16:24:28 UTC 2024 aarch64 GNU/Linux

@pelwell
Copy link
Contributor Author

pelwell commented Sep 17, 2024

Did you reboot after the install? Yes - I can see that you did.

There's another build with more reversions available via sudo rpi-update pulls/6369.

@lgeitner
Copy link

I ran sudo rpi-update pulls/6369 then sudo reboot

The meteopi is now working!

I would like to allow the meteopi to run overnight and check for any errors from the meteopi.
I will let you know if there are any errors or problems sometime tomorrow.

Thank You!

output from the command uname- a

pi@cumulusmxkitchen:~ $ uname -a
Linux cumulusmxkitchen 6.6.51-v8-16k+ #1 SMP PREEMPT Tue Sep 17 19:44:50 UTC 2024 aarch64 GNU/Linux
pi@cumulusmxkitchen:~ $

@lgeitner
Copy link

I just noticed when I ran the command dmesg there is the following error in red

[ 11.585942] pl011-axi 1f00030000.serial: unable to pause DMA transfer

pelwell added a commit to pelwell/linux that referenced this issue Sep 18, 2024
Some recent DMA changes have led to data loss in UART0 on Pi 5. It also
seems that even prior to these changes there was a problem with aborted
transfers.

As this is the only RP1 UART configured for DMA, it is better to remove
the DMA usage until it is shown to be reliable.

Link: raspberrypi#6365

Signed-off-by: Phil Elwell <[email protected]>
pelwell added a commit that referenced this issue Sep 18, 2024
Some recent DMA changes have led to data loss in UART0 on Pi 5. It also
seems that even prior to these changes there was a problem with aborted
transfers.

As this is the only RP1 UART configured for DMA, it is better to remove
the DMA usage until it is shown to be reliable.

Link: #6365

Signed-off-by: Phil Elwell <[email protected]>
@pelwell
Copy link
Contributor Author

pelwell commented Sep 18, 2024

sudo rpi-update pulls/6371 will get you the release candidate patch. It simply disables DMA for UART0 until it can be shown to be reliable.

@lgeitner
Copy link

I let the meteopi run overnight, there were no errors! Its working as expected!

Thank you again for your help!

I will try sudo rpi-update pulls/6371 sometime in the next couple hours and will let you know if I have any issues with it.

Thanks,
Lance

@lgeitner
Copy link

I ran the command sudo rpi-update pulls/6371 then sudo reboot.

The meteopi is working fine and the error "pl011-axi 1f00030000.serial: unable to pause DMA transfer" no longer appears in dmesg.

the output from uname -a is as follows:

pi@cumulusmxkitchen:~ $ uname -a
Linux cumulusmxkitchen 6.6.51-v8-16k+ #1 SMP PREEMPT Wed Sep 18 16:24:48 UTC 2024 aarch64 GNU/Linux
pi@cumulusmxkitchen:~ $

Thank you again for your help!

@pelwell
Copy link
Contributor Author

pelwell commented Sep 18, 2024

Thank you for reporting it - shipping that kernel in an image would have been embarrassing.

pelwell added a commit to pelwell/linux that referenced this issue Sep 19, 2024
In order to avoid losing residue bytes when a receive is terminated
early, set the destination width to single bytes.

Link: raspberrypi#6365

Signed-off-by: Phil Elwell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants