半個(gè)小時(shí)搭建自己的實(shí)時(shí)監(jiān)控系統(tǒng)

首先給直觀的看看監(jiān)控效果圖：


image.png

數(shù)據(jù)流架構(gòu)如下所示，通過(guò)Flume采集日志數(shù)據(jù)，并寫(xiě)入到kafka中，F(xiàn)link讀取kafka數(shù)據(jù)經(jīng)過(guò)處理后再次放入到kafka中，監(jiān)控頁(yè)面通過(guò)websocket監(jiān)聽(tīng)kafka中數(shù)據(jù)實(shí)現(xiàn)實(shí)時(shí)的數(shù)據(jù)顯示。


image.png

整體技術(shù)框架基于ruoyi單機(jī)版本搭建
新增加的文件如下：


image.png

第一步先啟動(dòng)Flume，F(xiàn)lume監(jiān)聽(tīng)文件，我這里通過(guò)tail命令監(jiān)聽(tīng)文件新寫(xiě)入的內(nèi)容

./flume-ng agent -c /Users/dbq/Documents/middleware/flume/master/apache-flume-1.9.0-bin/conf -f /Users/dbq/Documents/middleware/flume/master/apache-flume-1.9.0-bin/conf/kafka.conf -n a1 -Dflume.root.logger=INFO,console

配置文件如下，F(xiàn)lume實(shí)時(shí)監(jiān)控文件數(shù)據(jù)，并寫(xiě)入到kafka test 主題中

a1.sources = r1
a1.sinks = k1
a1.channels = c1
#對(duì)于source的配置描述 監(jiān)聽(tīng)文件中的新增數(shù)據(jù) exec
a1.sources.r1.type = exec
a1.sources.r1.command  = tail -F /Users/d/Documents/middleware/flume/data/log.00
a1.sources.ri.shell = /bin/sh -c
#對(duì)于sink的配置描述 使用kafka做數(shù)據(jù)的消費(fèi)
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = test
a1.sinks.k1.brokerList = 127.0.0.1:9092,127.0.0.1:9092,127.0.0.1:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 5
#對(duì)于channel的配置描述 使用內(nèi)存緩沖區(qū)域做數(shù)據(jù)的臨時(shí)緩存
a1.channels.c1.type = memory

#通過(guò)channel c1將source r1和sink k1關(guān)聯(lián)起來(lái)
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

第二步：系統(tǒng)啟動(dòng)的時(shí)候監(jiān)聽(tīng)kafka topic，并通過(guò)Flink進(jìn)行流式計(jì)算，Sink負(fù)責(zé)將處理后的數(shù)據(jù)輸出到外部系統(tǒng)中。

@Component
public class Runner implements CommandLineRunner {
    @Override
    public void run(String... args) throws Exception {
        System.out.println("--------------");
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<>("test", new SimpleStringSchema(), getProperties());
        DataStream<String> dataStream = env.addSource(consumer);

        //模擬業(yè)務(wù)過(guò)程流式處理
        DataStream<String> after = dataStream.map((MapFunction<String, String>) s -> {
            MonitorObject mo = getMonitorObject(s);
            return JSON.toJSONString(mo);
        });
        after.addSink(new MySink());
        env.execute("spring flink demo");
    }

    /**
     * 模擬二次處理
     */
    private static MonitorObject getMonitorObject(String s) {
        MonitorObject mo = JSON.toJavaObject(JSON.parseObject(s), MonitorObject.class);
        mo.setOil(mo.getOil() % 2);
        mo.setSpeed(mo.getSpeed()%2);
        return mo;
    }

    private Properties getProperties() {
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "localhost:9092");
        properties.setProperty("zookeeper.connect", "localhost:3181");
        properties.setProperty("group.id", "flink-group");
        properties.setProperty("auto.offset.reset", "latest");
        properties.setProperty("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        properties.setProperty("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        return properties;

    }
}

第三步：sink中講數(shù)據(jù)重新寫(xiě)入到kafka中，這里重寫(xiě)寫(xiě)入到kafka目的是起到平滑推送數(shù)據(jù)到前端頁(yè)面的效果，也方便以廣播的方式推送到其他業(yè)務(wù)系統(tǒng)，其他業(yè)務(wù)系統(tǒng)只需要訂閱test_after主題，就可以獲得Flink處理之后的數(shù)據(jù)

@Slf4j
@Component
public class MySink extends RichSinkFunction<String> {
    private AnnotationConfigApplicationContext ctx;

    private final static String topic = "test_after";

    public MySink() {
        log.info("mySink new");
    }

    @Override
    public void open(Configuration paramters) {
        this.ctx = new AnnotationConfigApplicationContext(Config.class);
        log.info("my sink open");
    }

    @Override
    public void invoke(String value, Context context) throws Exception {
        log.info("[flink監(jiān)控kafka數(shù)據(jù)]：{}", value);

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        List<String> data = new ArrayList<>();
        data.add(value);

        DataStreamSource<String> source = env.fromCollection(data);
        FlinkKafkaProducer<String> producer = new FlinkKafkaProducer<>(
                topic,
                (KafkaSerializationSchema<String>) (element, timestamp) -> new ProducerRecord<>(
                        topic,
                        element.getBytes()
                ),
                getProperties(),
                FlinkKafkaProducer.Semantic.NONE
        );
        //重新寫(xiě)入到kafka
        source.addSink(producer);
        env.execute();
    }

    @Override
    public void close() {
        ctx.close();
        log.info("my sink close");
    }

    private Properties getProperties() {
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "localhost:9092");
        properties.setProperty("zookeeper.connect", "localhost:3181");
        properties.setProperty("group.id", "flink-group");
        properties.setProperty("auto.offset.reset", "latest");
        return properties;

    }

第四步：監(jiān)聽(tīng)kafka，并通過(guò)websocket推送到前端頁(yè)面

@Component
@Slf4j
public class KafkaConsumer {

    @KafkaListener(groupId = "3",topics = "test_after")
    public void listen(String msg){
        System.out.println("====================> " + msg);
        MonitorObject mo = JSON.toJavaObject(JSON.parseObject(msg), MonitorObject.class);
        WebSocketUsers.sendMessageToUsersByText(mo);
    }
}

其他代碼
WebSocketConfig.java

@Configuration
public class WebSocketConfig
{
    @Bean
    public ServerEndpointExporter serverEndpointExporter()
    {
        return new ServerEndpointExporter();
    }
}

WebSocketServer.java

@Component
@ServerEndpoint("/websocket/message")
public class WebSocketServer
{
    /**
     * WebSocketServer 日志控制器
     */
    private static final Logger LOGGER = LoggerFactory.getLogger(WebSocketServer.class);

    /**
     * 默認(rèn)最多允許同時(shí)在線人數(shù)100
     */
    public static int socketMaxOnlineCount = 100;

    private static Semaphore socketSemaphore = new Semaphore(socketMaxOnlineCount);

    /**
     * 連接建立成功調(diào)用的方法
     */
    @OnOpen
    public void onOpen(Session session) throws Exception
    {
        boolean semaphoreFlag = false;
        // 嘗試獲取信號(hào)量
        semaphoreFlag = SemaphoreUtils.tryAcquire(socketSemaphore);
        if (!semaphoreFlag)
        {
            // 未獲取到信號(hào)量
            LOGGER.error("\n 當(dāng)前在線人數(shù)超過(guò)限制數(shù)- {}", socketMaxOnlineCount);
//            WebSocketUsers.sendMessageToUserByText(session, "當(dāng)前在線人數(shù)超過(guò)限制數(shù)：" + socketMaxOnlineCount);
            session.close();
        }
        else
        {
            // 添加用戶
            WebSocketUsers.put(session.getId(), session);
            LOGGER.info("\n 建立連接 - {}", session);
            LOGGER.info("\n 當(dāng)前人數(shù) - {}", WebSocketUsers.getUsers().size());
//            WebSocketUsers.sendMessageToUserByText(session, "連接成功");
        }
    }

    /**
     * 連接關(guān)閉時(shí)處理
     */
    @OnClose
    public void onClose(Session session)
    {
        LOGGER.info("\n 關(guān)閉連接 - {}", session);
        // 移除用戶
        WebSocketUsers.remove(session.getId());
        // 獲取到信號(hào)量則需釋放
        SemaphoreUtils.release(socketSemaphore);
    }

    /**
     * 拋出異常時(shí)處理
     */
    @OnError
    public void onError(Session session, Throwable exception) throws Exception
    {
        if (session.isOpen())
        {
            // 關(guān)閉連接
            session.close();
        }
        String sessionId = session.getId();
        LOGGER.info("\n 連接異常 - {}", sessionId);
        LOGGER.info("\n 異常信息 - {}", exception);
        // 移出用戶
        WebSocketUsers.remove(sessionId);
        // 獲取到信號(hào)量則需釋放
        SemaphoreUtils.release(socketSemaphore);
    }

    /**
     * 服務(wù)器接收到客戶端消息時(shí)調(diào)用的方法
     */
    @OnMessage
    public void onMessage(String message, Session session)
    {
        String msg = message.replace("你", "我").replace("嗎", "");
        WebSocketUsers.sendMessageToUserByText(session, msg);
    }
}

WebSocketUsers.java

/**
 * websocket 客戶端用戶集
 * 
 * @author ruoyi
 */
public class WebSocketUsers
{
    /**
     * WebSocketUsers 日志控制器
     */
    private static final Logger LOGGER = LoggerFactory.getLogger(WebSocketUsers.class);

    /**
     * 用戶集
     */
    private static Map<String, Session> USERS = new ConcurrentHashMap<String, Session>();

    /**
     * 存儲(chǔ)用戶
     *
     * @param key 唯一鍵
     * @param session 用戶信息
     */
    public static void put(String key, Session session)
    {
        USERS.put(key, session);
    }

    /**
     * 移除用戶
     *
     * @param session 用戶信息
     *
     * @return 移除結(jié)果
     */
    public static boolean remove(Session session)
    {
        String key = null;
        boolean flag = USERS.containsValue(session);
        if (flag)
        {
            Set<Map.Entry<String, Session>> entries = USERS.entrySet();
            for (Map.Entry<String, Session> entry : entries)
            {
                Session value = entry.getValue();
                if (value.equals(session))
                {
                    key = entry.getKey();
                    break;
                }
            }
        }
        else
        {
            return true;
        }
        return remove(key);
    }

    /**
     * 移出用戶
     *
     * @param key 鍵
     */
    public static boolean remove(String key)
    {
        LOGGER.info("\n 正在移出用戶 - {}", key);
        Session remove = USERS.remove(key);
        if (remove != null)
        {
            boolean containsValue = USERS.containsValue(remove);
            LOGGER.info("\n 移出結(jié)果 - {}", containsValue ? "失敗" : "成功");
            return containsValue;
        }
        else
        {
            return true;
        }
    }

    /**
     * 獲取在線用戶列表
     *
     * @return 返回用戶集合
     */
    public static Map<String, Session> getUsers()
    {
        return USERS;
    }

    /**
     * 群發(fā)消息文本消息
     *
     * @param message 消息內(nèi)容
     */
    public static void sendMessageToUsersByText(Object message)
    {
        Collection<Session> values = USERS.values();
        for (Session value : values)
        {
            sendMessageToUserByText(value, message);
        }
    }

    /**
     * 發(fā)送文本消息
     *
     * @param session 自己的用戶名
     * @param message 消息內(nèi)容
     */
    public static void sendMessageToUserByText(Session session, Object message)
    {
        if (session != null)
        {
            try
            {
                session.getBasicRemote().sendText(JSON.toJSONString(message));
            }
            catch (IOException e)
            {
                LOGGER.error("\n[發(fā)送消息異常]", e);
            }
        }
        else
        {
            LOGGER.info("\n[你已離線]");
        }
    }
}

前端代碼

var url = "ws://127.0.0.1:80/websocket/message";
        var ws = new WebSocket(url);
        ws.onopen = function() {
            $('#text_content').append('已經(jīng)打開(kāi)連接!' + '\n');
        }
        ws.onmessage = function(event) {
            console.log(event.data)
            var obj = JSON.parse(event.data);
            gaugeChart.setOption({
                series : [
                    {
                        name: '速度',
                        data: [{value: obj.speed}]
                    },
                    {
                        name: '轉(zhuǎn)速',
                        data: [{value: obj.rotate_speed}]
                    },
                    {
                        name: '油耗',
                        data: [{value: obj.oil}]
                    }
                ]
            })
        }

模擬數(shù)據(jù)的持續(xù)生成，這里每秒鐘生成一條數(shù)據(jù)，以json格式寫(xiě)入到日志文件中

while true
do
 echo "{\"speed\":$((RANDOM %220)),\"rotate_speed\":$((RANDOM %7)),\"oil\":$((RANDOM %3))}" >> log.00
 sleep 1
done
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書(shū)系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容