How is the real-time number of visitors on the site achieved?

Readers who frequently visit my site should have noticed the display of the current number of online users at the bottom of the page. Not long ago, I also added a ranking of the number of people currently reading specific articles, and on the left side of a specific article, I added a timeline to show the reading progress of all readers currently reading that article.

If you have seen my site's open-source projects, you probably know that this site is powered by Mix Space. Since a server is required for operation, I basically do not rely on services provided by other platforms; anything that can be implemented on my own is done by myself. For example, for this feature, if you use a SaaS service, you can choose Liveblocks. Of course, this is not the focus of the discussion here, as the purpose of this article is to explain how to implement such a feature by yourself, both on the server side and the front end.

Design and Implementation#

Real-time user count statistics require long connections. Here, WebSocket can be used; of course, if you do not need particularly real-time data, polling can also be used. We will still use Socket.IO for implementation, which is a higher-level wrapper for WebSocket.

Let’s break down and analyze the above functionality.

Real-time User Count of the Site#

The total real-time user count is relatively easy to calculate. We can consider using all WebSocket connections to count the current number of online users.

If this method is used, then readers who are not connected to WebSocket will not be counted as online.

Since the site can be accessed without logging in, each visit will establish a WebSocket connection. If the same user opens multiple pages to access the site simultaneously, it will lead to duplicate counting of online users.

For the first point above, we will not consider it. For the second point, we can analyze it slightly and impose some restrictions in certain scenarios.

It is acceptable for each opened page to establish a WebSocket connection, but we need to deduplicate these connections for the same user.

The following methods can be adopted:

Deduplicate based on the IP of the same WebSocket connection (this may be too aggressive in a large internal network or CDN scenario).
Generate a unique SessionId for the connection on the client side and share it within the browser (Local Storage can be used).
Deduplicate based on the user's login status or locally recorded historical comment data.

In summary, we finally choose 2 and 3 as the final implementation.

https://cdn.jsdelivr.net/npm/@innei/[email protected]/excalidraw/realtime-socket/socket-client.json

So, on the front end, we need to generate a SessionId when there isn't one; if it exists in Local Storage, we reuse it.

import { customAlphabet } from 'nanoid'

const alphabet = `1234567890abcdefghijklmnopqrstuvwxyz`

const nanoid = customAlphabet(alphabet)
const defaultSessionId = nanoid(8)
const storageKey = buildNSKey('web-session')

const getSocketWebSessionId = () => {
  if (!isClientSide) {
    return ''
  }
  const sessionId = localStorage.getItem(storageKey)
  if (sessionId) return sessionId
  localStorage.setItem(storageKey, defaultSessionId)
  return defaultSessionId
}

First, we use nanoid to generate a default sessionId, and we need to pass the existing SessionId to the Socket Client. As for the third point regarding the Session Id, since it involves asynchronous login state, I can only choose to update the current Socket Client's Session Id later.

const client = io(GATEWAY_URL, {
      timeout: 10000,
      reconnectionDelay: 3000,
      autoConnect: false,
      reconnectionAttempts: 3,
      transports: ['websocket'],

      query: {
        socket_session_id: getSocketWebSessionId(),
      },
})

Here, the generated WebSessionId is passed through the query.

Important

The Socket.IO client will have a sid value, which is the unique id for each client; please do not confuse it with our generated session id.

On the server side, consume the passed sessionId.

 async handleConnection(socket: SocketIO.Socket) {
    const webSessionId =
      socket.handshake.query['socket_session_id'] ||
      // fallback sid
      socket.id

    await this.gatewayService.setSocketMetadata(socket, {
      sessionId: webSessionId,
    })
}

Here, I used setSocketMetadata to store metadata on the socket for later retrieval and updates. This metadata attached to the Socket is stored in Redis; you can refer to the specific implementation at https://github.com/mx-space/core/blob/c82cb8ff54dc7b98bc411e03de81ede932aa612b/apps/core/src/processors/gateway/gateway.service.ts.

Now, we implement the total user count statistics on the server side as follows:

async getCurrentClientCount() {
    const server = this.namespace.server

    const socketsMeta = await Promise.all(
      await server
        .of(`/${namespace}`)
        .fetchSockets()
        .then((sockets) => {
          return sockets.map((socket) =>
            this.gatewayService.getSocketMetadata(socket),
          )
        }),
    )
    return uniqBy(socketsMeta, (x) => x?.sessionId).length
}

When there are other WebSocket connections, broadcast the current user count to all Socket Clients.

async handleConnection(socket: SocketIO.Socket) {
  this.whenUserOnline()
}

whenUserOnline = async () => {
  this.broadcast(
    BusinessEvents.VISITOR_ONLINE,
    await this.sendOnlineNumber(),
  )
}

Update SessionId#

After the front-end user completes the login state, we need to update the SocketId to obtain a more accurate real-time user count.

https://cdn.jsdelivr.net/npm/@innei/[email protected]/excalidraw/realtime-socket/update-session-id.json

Some states may exist in Hooks, so we also encapsulate a Hook to obtain the final SocketId.

export const useSocketSessionId = () => {
  const user = useUser()
  const owner = useOwner()
  const ownerIsLogin = useIsLogged()

  return useMemo((): string => {
    const fallbackSid = getSocketWebSessionId()
    if (ownerIsLogin) {
      if (!owner) return fallbackSid
      return `owner-${owner.id}`
    } else if (user && user.isSignedIn) {
      return user.user.id.toLowerCase()
    }
    return fallbackSid
  }, [owner, ownerIsLogin, user])
}

Then emit the update event.

  const webSocketSessionId = useSocketSessionId()
  const previousWebSocketSessionIdRef = useRef(webSocketSessionId)

  const socketIsConnected = useSocketIsConnect()

  useEffect(() => {
    const previousWebSocketSessionId = previousWebSocketSessionIdRef.current
    previousWebSocketSessionIdRef.current = webSocketSessionId
    if (!socketIsConnected) return

    socketClient.emit(SocketEmitEnum.UpdateSid, {
      sessionId: webSocketSessionId,
    })
  }, [socketIsConnected, webSocketSessionId])

Now, on the server side, handle this event.

@SubscribeMessage('message')
async handleMessageEvent(
  @MessageBody() data: MessageEventDto,
  @ConnectedSocket() socket: SocketIO.Socket,
) {
  const { payload, type } = data
  switch (type) {
    case SupportedMessageEvent.UpdateSid: {
      const { sessionId } = payload as { sessionId: string }
      if (sessionId) {
        await this.gatewayService.setSocketMetadata(socket, { sessionId })
        this.whenUserOnline() // broadcast all client online count
      }
    }
  }
}

Real-time Reading Count of Articles#

For each article, we need to count the current number of online readers, which can leverage the Room feature of Socket.IO. If Socket.IO is not used, a custom Channel can be implemented, which means splitting a Socket into multiple Scopes, where each Scope can subscribe to different types of message pushes.

Thus, each article corresponds to a Room, and Socket Clients in the same Room can receive messages from each other. When readers scroll the page, emit an event, and on the server side, broadcast the reading progress of a reader to the entire Room.

Upon entering the article, immediately send a request to join the Room, and when switching articles or closing the page, exit the Room.

https://cdn.jsdelivr.net/npm/@innei/[email protected]/excalidraw/realtime-socket/reading-room.json

On the server side, first implement the handling of Join and Leave events.

@SubscribeMessage('message')
async handleMessageEvent(
  @MessageBody() data: MessageEventDto,
  @ConnectedSocket() socket: SocketIO.Socket,
) {
  const { payload, type } = data

  switch (type) {
    case SupportedMessageEvent.Join: {
      const { roomName } = payload as { roomName: string }
      if (roomName) {
        socket.join(roomName)
      }
      break
    }
    case SupportedMessageEvent.Leave: {
      const { roomName } = payload as { roomName: string }
      if (roomName) {
        socket.leave(roomName)
        const socketMeta = await this.gatewayService.getSocketMetadata(socket)
        if (socketMeta.presence) {
          this.webGateway.broadcast(
            BusinessEvents.ACTIVITY_LEAVE_PRESENCE,
            {
              identity: socketMeta.presence.identity,
              roomName,
            },
            {
              rooms: [roomName],
            },
          )
          handlePresencePersistToDb(socket)
        }
        const roomJoinedAtMap = await this.getSocketRoomJoinedAtMap(socket)
        delete roomJoinedAtMap[roomName]
        await this.gatewayService.setSocketMetadata(socket, {
          roomJoinedAtMap,
        })
      }
      break
    }
  }
}

RoomName can be generated based on the article's id to create a unique key. Then use socket.join() to join a room, which is an internal implementation of Socket.IO that we do not need to concern ourselves with; we only need to attach or modify some metadata or persist some data to the DB during Join or Leave.

Next, on the front end, when the reader scrolls the page, we will send the current reading progress to the server through an interface, and then the server will broadcast the reading status of the user who made the request to all Socket Clients in that Room. Therefore, when requesting the interface, we need to send the current RoomName, socket.sid, the front-end generated sessionId, and the reading position.

https://cdn.jsdelivr.net/npm/@innei/[email protected]/excalidraw/realtime-socket/presence-broadcast.json

Assuming the data structure sent from the front end is as follows.

interface Persence {
    roomName: string;
    position: number;
    identity: string;
    sid: string;
    displayName?: string;
}

The key implementation on the back end is:

async updatePresence(data: UpdatePresenceDto) {
  const roomName = data.roomName

  if (!isValidRoomName(roomName)) {
    throw new BadRequestException('invalid room_name')
  }
  const roomSockets = await this.webGateway.getSocketsOfRoom(roomName)

  const presenceData: ActivityPresence = {
    ...data,

    operationTime: data.ts,
    updatedAt: Date.now(),
    connectedAt: +new Date(socket.handshake.time),
  }
  const roomJoinedAtMap =
    await this.webGateway.getSocketRoomJoinedAtMap(socket)

  Reflect.set(serializedPresenceData, 'joinedAt', roomJoinedAtMap[roomName])

  this.webGateway.broadcast(
    BusinessEvents.ACTIVITY_UPDATE_PRESENCE,
    serializedPresenceData,
    {
      rooms: [roomName],
    },
  )

  await this.gatewayService.setSocketMetadata(socket, {
    presence: presenceData,
  })

  return serializedPresenceData
}

When the interface is reached, the server will broadcast the reading status of the user who made the request.

Current Article Reading Ranking on the Site#

This feature is actually built on the foundation of the above, so once we complete the previous feature, this one becomes straightforward. We just need to get all existing Rooms and then get the number of Socket Clients in each Room.

This is easy to understand, so I won't elaborate further.

Conclusion#

All the code above is open source. The specific implementation can be found at: mx-space/core, Shiro

This article is synchronized and updated to xLog by Mix Space. The original link is https://innei.in/posts/dev-story/how-is-the-real-time-headcount-at-the-site-achieved