My July 2020 experience with Sametime 11.0 FP1
My experience installing Sametime 11.0 FP1.
July 13, 2020: only 2 days before the IBM Cloud shutdown and my customer that has been using Samtime in the cloud needs to get moved to a new on-prem Sametime (ST), like now.
I had attended and HCL webinar that said you can install Domino 11.0.1 and then Sametime 11.0 FP1 (no reason to install Sametime 11 first).
HCL Webinar
Well I found reasons.
I started with a new VM.
Installed CentOS 8
The documentation looked promising: HCL Sametime 11
Installed MongoDb, just as the instructions indicated. No problems.
Installed Domino 11.0.1. No issues.
Configured Domino.
Made sure I could get to the server via http with a web browser (this Sametime server is behind our firewall so I don't have SSL). We are not going to support Sametime on mobile devices, so not Proxy Server install, and we have only a few people so no Sametime MUX installation. This is a pretty plain install and I think it could be a common install for many companies under 500 users.
Install Dan Nashed's scripts. The best way to run Domino on Linux.
Now the Fun starts. Let's install Sametime.
tar -xvf Sametime_11.0_FP1_CommunityServer_Linux64.tar
created the installer.properties files in the same directory as the unzipped Sametime install.bin file according to the online documentation.
This was the content of my installer.properties file. Note: I'm using the Domino directory on the Sametime "chat" server instead of LDAP in this situtation.
# This file was built by the Replay feature of InstallAnywhere.
# It contains variables that were set by Panels, Consoles or Custom Code.
DataInstallDir=/local/notesdata
UNIX_UserName=notes
UNIX_GroupName=notes
UNIX_ServerName=chat.mydomain.com
#Directory Selection
#-------------------
DIRECTORY_TYPE_DOMINO=1
DIRECTORY_TYPE_LDAP=0
LDAP_SERVER=
LDAP_PORT=
ADVANCED_LDAP_CONFIG=0
ST_BRANDING_INFO=entry
#Install
#-------
-fileOverwrite_/tmp/156305.tmp/stnotesinstallutilities_linux64=Yes
-fileOverwrite_/tmp/156305.tmp/stnotesinstallutilities.jar=Yes
-fileOverwrite_/tmp/156305.tmp/stnotesinstallutilities.dll=Yes
-fileOverwrite_/tmp/156305.tmp/stinstallutilities.dll=Yes
-fileOverwrite_/tmp/156305.tmp/vcredist_x64.exe=Yes
-fileOverwrite_/tmp/156305.tmp/UnixSetNotesFieldValue.sh=Yes
-fileOverwrite_/local/notesdata/_HCL Sametime Server 11.0.0_installation/Change HCL Sametime Server 11.0.0 Installation.lax=Yes to All
Now run the silent installer because I didn't install any graphics on this linux server. ./install.bin -i silent
Took about 20 seconds. There is no response from it, it simply completes. I check the files as indicated in the documentation to check if successful and it seems like it was.
Start Domino. The staddin task never fired off. Didn't even attempt to.
I noticed a problem that the directory assistance database (da.nsf) and our second NAB: vendors.nsf weren't replicated to the ST server. I fixed that.
The titles of the Sametime related database in the notesdata directory looked very odd. How does this happen?
Here is a list of files as seen in the Domino Admin client. Notice the Database title and file names. This can't be good. There aren't really duplicates, some are links to correct for STautht.nsf versus stauth1.nsf because linux is case sensitive.
| Title | Filename |
|---|---|
| You cannot do a remote queue put to a pre-R5 server | stauths.nsf |
| You cannot administer Enterprise Directories on a pre-R5 server | stautht.nsf |
| You cannot administer Enterprise Directories on a pre-R5 server | stautht.nsf |
| Sametime Center | stcenter.nsf |
| Network protocol error: message from server is too large | stconf.nsf |
| Accelerated replica creation cannot be used with an encrypted database | stconfig.nsf |
| Accelerated replica creation cannot be used with an encrypted database | stconfig.nsf |
| Invalid profile | stcs.nsf |
| Cannot close a database within an NSFSearchStart - NSFSearchStop loop that was opened outside of the loop | stdomino.nsf |
| LookupExtended on server %a | stlog.nsf |
| LookupExtended on server %a | stlog.nsf |
| Invalid VJournal property found | stnamechange.nsf |
| Sametime Name Change | stnamechange.ntf |
| Sametime Offline Messages | stofflinemessages.nsf |
| Sametime Offline Messages | stofflinemessages.ntf |
| Sametime Resources | stsrc.nsf |
| Extended access controls are enabled on this database. You must modify the database on a version 6 or later Domino server. | vpuserinfo.nsf |
I read Gabby's post
Decided I had better try and put on ST 11.0 first.
I was able to uninstall because when ST installed it created the un-install directory and files. Shut down Domino and I ran this:/local/notesdata/_HCL\ Sametime\ Server\ 11.0\ FP1_installation/Change\ HCL\ Sametime\ Server\ 11.0\ FP1\ Installation
Installed ST 11.0 Community Server, I used the same installer.properties file. as I used for 11.0.1.
Then ran the ST 11.0 FP1 install.
Started the server, this time staddin attempted to run, but I saw errors:
HTTP Server: Error loading Web SSO Cookie Names Configuration 'LtpaToken' (Single Sign-On configuration is invalid).
I don't know if this is still necessary, but in older version of ST, we couldn't use "Internet Site documents". We had to use the old way and configure HTTP and Domino web on the Server doc. So I switched back to Internet Site Docs 'disabled', then I did the old trick of copying the SSO token doc from Internet sites (that included my new ST server) and remove the organization.
Restart, the loading Web SSO Cookie issue went away but no STaddin.
So, I decided to uninstall and go with ST 11. I deleted all the Samtime NSF files (because many had messed up titles), then shut down Domino. I unistalled ST again and installed ST11. Now stAddin is launching but throwing lots of errors. The database titles are still messed up.
I still don't have a stchatlogging.nsf file that Gabby talks about. Even when I finally got this working (yes, I think I eventually did), I do not have stchatlogging.nsf
In stconfig.nsf, I set chat logging to relaxed as indicated in the documentation.
I installed ST11 FP1 over the top.
staddin still isn't working. load staddin does nothing.
So, I go back and install ST11 over the top. staddin is loading, and chat seems to be working. Good to go, or so I think.
The next day I have people logging on to it. about 5 people.
My server admin notices a CPU spike. It goes to 100% and stays there. I check linux with the 'top' command and th stchatlogging services is running at 300+% of CPU (not sure how it is running above a 100, but it is).
tell staddin quit
cpu goes to normal
load staddin
the stchatlogging service is small, 1%, goes up and down a little. But over time the CPU keeps increasing and when half a day is gone it's pegged again. When the Domino server CPU is pegged, it doesn't like to do much, such as replicate names.nsf. The replica task just fails. For a couple days, I keep the ST IM functionality by unloading ST and reloading it to reset the threads that stchatlogging is spawning. 'tell staddin q', then 'load staddin'
So, it is over to Darren Duke's post (I was really hoping I wasn't going to need to go this deep into linux. I know only enough linux to be dangerous).
Well it actually wasn't too bad once I understood a bit. To fix missing links and add libraries that weren't there, these we my steps, almost all the same as Darren.
first install the locate bins:
yum install mlocate
updatedb (must update the db for locate to work)
locate libstchatloggingmongodb.so
returns:
/opt/hcl/domino/notes/11000100/linux/libstchatloggingmongodb.so
run: [root@st lib64]# ldd /opt/hcl/domino/notes/11000100/linux/libstchatloggingmongodb.so
returns:
linux-vdso.so.1 (0x00007ffe2e7e5000)
libmongoclient.so => not found
libcrypto.so.1.0.0 => not found
libssl.so.1.0.0 => not found
libubq.so => not found
libwinux.so => not found
libstts.so => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f95b854b000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f95b8347000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f95b7fb2000)
libm.so.6 => /lib64/libm.so.6 (0x00007f95b7c30000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f95b7a18000)
libc.so.6 => /lib64/libc.so.6 (0x00007f95b7655000)
/lib64/ld-linux-x86-64.so.2 (0x00007f95b89c2000)
Looks like it is missing a lot of libraries
Here is an example of how I dealt with each line
locate libmongoclient.so
returns: /opt/hcl/domino/notes/11000100/linux/libmongoclient.so
ln -s /opt/hcl/domino/notes/11000100/linux/libmongoclient.so /usr/lib64/libmongoclient.so
ldd /opt/hcl/domino/notes/11000100/linux/libstchatloggingmongodb.so
returns:
linux-vdso.so.1 (0x00007fffaa929000)
libmongoclient.so => /lib64/libmongoclient.so (0x00007f06f0859000)
libcrypto.so.1.0.0 => not found
libssl.so.1.0.0 => not found
libubq.so => not found
libwinux.so => /lib64/libwinux.so (0x00007f06f05f5000)
libstts.so => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f06f03d5000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f06f01d1000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f06efe3c000)
libm.so.6 => /lib64/libm.so.6 (0x00007f06efaba000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f06ef8a2000)
libc.so.6 => /lib64/libc.so.6 (0x00007f06ef4df000)
libssl.so.10 => not found
libcrypto.so.10 => not found
libboost_system.so.1.63.0 => not found
libboost_thread.so.1.63.0 => not found
libboost_regex.so.1.63.0 => not found
librt.so.1 => /lib64/librt.so.1 (0x00007f06ef2d6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f06f0e09000)
That seemed to help. I go through the locate of each remaining 'not found' files and wind up running these commands on my system
ln -s /opt/hcl/domino/notes/11000100/linux/STOpenSSL/libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
ln -s /opt/hcl/domino/notes/11000100/linux/STOpenSSL/libssl.so.1.0.0 /usr/lib64/libssl.so.1.0.0
ln -s /opt/hcl/domino/notes/11000100/linux/libubq.so /usr/lib64/libubq.so
ln -s /opt/hcl/domino/notes/11000100/linux/libstts.so /usr/lib64/libstts.so
ln -s /opt/hcl/domino/notes/11000100/linux/libboost_system.so.1.63.0 /usr/lib64/libboost_system.so.1.63.0
ln -s /opt/hcl/domino/notes/11000100/linux/libboost_thread.so.1.63.0 /usr/lib64/libboost_thread.so.1.63.0
ln -s /opt/hcl/domino/notes/11000100/linux/libboost_regex.so.1.63.0 /usr/lib64/libboost_regex.so.1.63.0
I could not 'locate' libssl.so.10 or libcrypto.so.10 (same as Darren)
so I ran this: yum install compat-openssl10
*I did not need to create links for libssl.so.10 or libcrypto.so.10 after installing compat-openssl10.
Sametime is running again. I don't know if the CPU is going to max out again. I will find out tomorrow.
I wonder what version of ST I'm running now.
I look at /local/notesdata/domino/html/sametime/buildinfo.txt and I see:
Wed, Dec 18, 2019 4:06:34 PM
release
English
SAMETIME10.0_20191218-1509_WIN32AIXSOLLIN_WESTFORD
IMWC
Date=Wed, Dec 18, 2019 4:06:34 PM
Type=release
Language=English
Version=SAMETIME10.0_20191218-1509_WIN32AIXSOLLIN_WESTFORD
Flavor=IMWC
I know I didn't install ST 10. It is a brand new box. I installed ST 11.0 and 11.0 FP1 (not sure which one I did last). It seems HCL may not have updated these files.
The /local/notesdata/domino/html/sametime/buildinfo-unix.txt file has this:
Wed Dec 18 15:20:26 EST 2019
RELEASE
ltn-ocs-04.prod.hclpnp.com
Linux
SAMETIME10.0_20191218-1509_WIN32AIXSOLLIN_WESTFORD
x86_64
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
It is now the 'next day', 22 July 2020. The 50 users are logged on to Sametime and the CPU is just fine. The stchatlogging task is not burning up CPU
Lessons learned
- HCL is no better than IBM currently on documentation
- HCL needs to spend more time testing the installation process
- The linux installation scripts are obviously broken
- I would have never got ST running without the help of Darren Duke and his willingness to fight through linux stuff and do HCL's job for them
- As a consultant, you will spend way more time installing ST than you estimated (the first time)
- The Sametime webinar I attended seemed to be really good, but the information put out was inaccurate
- I still don't think I have FP1 installed and I don't see how I can verify the version I'm running
- The mongodb install went very smooth. The documentation was spot on as long as you do everything exactly as written. It's all a black-box to me
- If Sametime was a little more straight foward to install, without all the work-around, I think all Domino customers would have to install it, but right now attempting an upgrade on your own would be risky