Workflow when updating CEF version

[cztomczak]

These instructions are for updating CEF version in CEF Python. When building CEF from sources follow all steps. When using prebuilt binaries from Spotify Automated Builds then the steps for updating CEF are much easier, you can omit steps 1-5 can in such case.

Notes:

• As of April 2017 up to 28 GB of memory may be used (RAM + virtual). Each year Chromium memory requirements go higher so I recommend reserving more.
• As of May 2018 about 13 GB of internet transfer may be used
• As of May 2018 CEF seems to raise exception when patching fails, so no need to manually check that (todo: update instructions below)

Important changes in upstream CEF

There is the CEF Announce mailing list where you can find announcements of any new important changes in CEF.

Update CEF version in cefpython

1. Make sure you have enough RAM to build Chromium. If you don't have enough RAM build may fail at the end during linking of libcef library. Also due to not enough RAM the build may slow down your computer so significantly that user interface won't be responding at all (noticed on Linux during last step of linking libcef which can take hours). You can use virtual memory if you have a fast SSD disk. If you installed Linux without swap then see "Linux swap" notes at the end that provide instructions on how to re-enable swap.

   • When not having enough RAM and using swap, to speed up last steps of linking libcef.so try using --ninja-jobs 1 flag set only one ninja job. If using more ninja jobs more of them will use slow swap memory.

2. It is recommended to update patches using upstream cef/tools/patch_updater.py tool. If patch_updater.py fails then you need to manually resolve the conflicts and re-run with the --resave flag.

   1. Operation to be done in in chromium/cef/ . CEF expects chromium checkout in parent directory. However since you first need to update Chromium using automate.py you have no way to fix patches first. So you need to run it and see in the console if there are errors when applying patches and if so, terminate it, go back to this step 1 and then re-run automate.py in step 3.
   2. After cef/tools/patch_updater.py was ran and succeeded, copy chromium/src/cef/patch/patches/issue*.patch patches to cefpython/patches/ and overwrite.

3. To resolve conflict in patch manually copy the patch to appropriate CEF or Chromium directory:

   • Apply the patch with patch -p0 < issue251.patch command. Conflicts will be saved to ".rej" files.
   • Delete the patch file and generate a new patch file with git diff --no-prefix --relative > issueXXX.patch command
   • Update the patch in the patches/ directory

4. Create the build directory, enter it and run the automate.py tool. Somewhere at the beginning of execution you should see messages about patches being applied, make sure they applied successfully - this is explained in step 3).

   cd cefpython/
   mkdir build
   cd build/
   python ../tools/automate.py --build-cef --ninja-jobs 6 --cef-branch 2883 --force-chromium-update
   

   • The --force-chromium-update flag needs to be specified when Chromium version changed since last update. When just rebuilding project files you don't need to specify it, it will slow down the process.
   • If the --cef-branch flag isn't specified then CEF branch is taken from the src/version/cef_version_xxx.h file
   • There is a --no-cef-update flag, but there may be issues when using this flag due to CEF Issue 1825
     • If you make changes to CEF and want to rebuilt with these changes applied to CEF then specify the --no-cef-update flag
     • If CEF did not change or if build was interrupted and you want to continue building, you can use the ---no-cef-update flag so that CEF repo is not fetched again

4b. If there is such an error:

Command: python -- /home/cz/github/cefpython/build/chromium/src/build/config/linux/pkg-config.py
-s /home/cz/github/cefpython/build/chromium/src/build/linux/debian_wheezy_arm-sysroot -a arm
gmodule-2.0 gtk+-3.0 gthread-2.0
Returned 1.

Then remove then remove the chromium/src/build/linux/debian_wheezy_arm-sysroot directory. See explanation here.

4c. If there is an error on one of last steps during linking of libcef library it might be caused by not enough RAM memory.

5. Make sure that cefpython patches were applied successfully. Currently when patch fails, CEF will continue building. You need to go through output and find messages about patches being applied, like below. In this case issue125 patch failed. Patches issue231 and issue251 succeeded. If you have problems finding these messages, after you've built new CEF, try running the automate.py script again (without the --force-chromium-update flag this time) and it should appear somewhere at the beginning.

   ...
   --> already patched  ../chrome/app/generated_resources.grd
   --> already patched  ../components/plugins/renderer/loadable_plugin_placeholder.cc
   --> already patched  ../components/plugins/renderer/webview_plugin.cc
   --> already patched  ../components/plugins/renderer/webview_plugin.h
   Reading patch file /home/cz/github/cefpython/build/chromium/src/cef/patch/patches/issue125.patch
   --> file is not patched - failed hunk: 1
   --> source file is different - ../net/http/http_cache_transaction.cc
   Reading patch file /home/cz/github/cefpython/build/chromium/src/cef/patch/patches/issue231.patch
   --> successfully patched ./include/capi/cef_path_util_capi.h
   --> successfully patched ./include/cef_path_util.h
   --> successfully patched ./libcef/browser/path_util_impl.cc
   --> successfully patched ./libcef_dll/libcef_dll.cc
   --> successfully patched ./libcef_dll/wrapper/libcef_dll_wrapper.cc
   Reading patch file /home/cz/github/cefpython/build/chromium/src/cef/patch/patches/issue251.patch
   --> successfully patched ./include/capi/cef_drag_data_capi.h
   --> successfully patched ./include/cef_drag_data.h
   --> successfully patched ./libcef/browser/osr/web_contents_view_osr.cc
   --> successfully patched ./libcef/common/drag_data_impl.cc
   --> successfully patched ./libcef/common/drag_data_impl.h
   --> successfully patched ./libcef_dll/cpptoc/drag_data_cpptoc.cc
   --> successfully patched ./libcef_dll/ctocpp/drag_data_ctocpp.cc
   --> successfully patched ./libcef_dll/ctocpp/drag_data_ctocpp.h
   

6. Update the "src/version/cef_version_xxx.h" file (xxx is platform) with the contents of the "build/chromium/src/cef/binary_distrib/cef_binary_xxx/include/cef_version.h" file

7. Update src/include/ headers with CEF repo's include/ directory:

   • ignore the capi/ directory
   • keep cefpython's modifications in cef_path_util.h done by Issue Discovery of the "icudtl.dat" file fails on Linux. Merge patch into upstream. #231 patch (see patches/)
   • if API changed in CEF then additional modifications may be required in cefpython's code

8. Compare and synchronize changes between these files:

   • Compare "cefpython/subprocess/print_handler_gtk.cpp" with "cef/tests/cefclient/browser/print_handler_gtk.cc". See the .patch file in subprocess/ dir for required differences. Compare also corresponding .h files.
   • Compare "cefpython/src/client_handler/dialog_handler_gtk.cpp" with "cef/tests/cefclient/browser/dialog_handler_gtk.cc". See the .patch file in client_handler/ dir for required differences. Compare also corresponding .h files.
   • Compare "cefpython/src/subprocess/main_message_loop*" files with "cef/tests/shared/browser/main_message_loop*" files.

Steps for making a release

1. Run tools/build_distrib.py script on all platforms
2. Upload wheel packages to PyPI (use Python 3 to avoid SSL issues):

python -m pip install twine
~/.pyenv/versions/3.6.1/bin/twine upload *.whl

Example commands

64-bit

python ../tools/automate.py --build-cef --ninja-jobs 8 --cef-branch 3359

32-bit

python ../tools/automate.py --build-cef --ninja-jobs 8 --cef-branch 3359 --x86

Rebuild manually.

cd cefpython/build/chromium/src/
ninja -v -j8 -Cout/Release_GN_x64 cefsimple chrome_sandbox
ninja -v -j8 -Cout/Release_GN_x86 cefsimple chrome_sandbox

Package manually. For 64-bit add --x64-build flag

cd cefpython/build/chromium/src/cef/tools/
./make_distrib.sh --allow-partial --ninja-build --no-archive

Create prebuilt binaries and libraries for 32-bit. Must be done on 32-bit Linux.

cd cefpython/build/
python ../tools/automate.py --prebuilt-cef --x86

Build libcef_dll_wrapper library on Mac when using binaries from Spotify Automated Builds:

cmake -G "Ninja" -DPROJECT_ARCH="x86_64" -DCMAKE_CXX_FLAGS="-stdlib=libc++" -DCMAKE_BUILD_TYPE=Release ..
ninja libcef_dll_wrapper

Linux swap

As of April 2017 Chromium can use up to 28 GB of RAM. If there is not enough RAM on Linux during last steps when linking libcef library and you didn't create swap partition (virtual memory) during Linux installation, then do these steps:

1. Download "gparted live CD" and burn on DVD
2. Boot from DVD and choose "Safe graphics" mode
3. Shrink partition from the end (from the right) - shrinking from the left can cause issues.
4. Create "linux-swap" partition from the unallocated space
5. Shutdown gparted.
6. Boot real OS. Assuming /dev/sda2 == swap partition
7. sudo mkswap /dev/sda2
8. sudo blkid /dev/sda2 --> to see UUID eg. 0d729c04-828e-4a0f-9bc2-44324e7fc3ee
9. sudo swapon -U UUID
10. To disable swap partition: sudo swapoff -U UUID

Other notes

• Get rid of Issue Discovery of the "icudtl.dat" file fails on Linux. Merge patch into upstream. #231 and Drag & drop in OSR mode (Kivy). Expose API on all platforms. #251 patches by integrating it in upstream. These two patches prevent being able to use prebuilt binaries from Spotify automated builds. Patches already sent, patch 231 needs to be reworked, patch 251 is still waiting for review. Pull requests created in upstream:
  • https://bitbucket.org/chromiumembedded/cef/pull-requests/66/override-paths-dir_exe-dir_module-on-linux/diff
  • https://bitbucket.org/chromiumembedded/cef/pull-requests/80/add-functions-to-support-image/diff