Bigdata- 01

 

 

아직까지 크게 막힌 부분이 없고, 여러 문서를 보면서 하느라 정리를 못했다. 하둡 설치 전까지 벌어진 일을 정리할 시간이 필요해 정리한다. 이 보스팅을 그대로 따라가면 복수의 라즈베리파이를 이용한 클러스터 구성과 Hadoop 하둡 설치, Spark 설치까지 문제없이 따라할 수 있도록 아주 친절하고 상세히 작성했다.

 

이게 1편이고 포스팅이 길어지면 작성하면서 나눌 생각이다. 계획에 Plan B가 있으면 실제fh 플랜 A를 달성할 확률은 크게 낮아진다. 계획을 세우지 말라는 이야기가 아니라 최선을 다하고 순간에는 흐름을 따라 대처하는 능력이 필요하다는 말이다. 플랜 A가 전부인 것처럼 행동한다.

 

이거 그대로 한다. 순서대로 하면 잘된다. 방화벽까지....

 

 

ifconfig 명령이 먹지 않으니 다음으로 설치

 

sudo apt install net-tools

 

ip 주소 확인 가능

 

ssh 서비스 참고하고 아래를 참고하여 서비스가 정상 실행되도록 한다.

 

https://fishpoint.tistory.com/9555

Unit sshd.service could not be found 에러 해결

 

 

https://winterbloooom.github.io/computer%20science/linux/2022/03/22/ssh.html

 

 

 

 

하고 나면 맨 아래 Putty를 사용해 원격 접속을 확인한다.

 

여기까지가 시작 전이다.

 

 

goodtech@goodtech-desktop:~$ ssh -V
OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024
goodtech@goodtech-desktop:~$ sudo systemctl status sshd
[sudo] password for goodtech: 
Unit sshd.service could not be found.
goodtech@goodtech-desktop:~$

 

 

 

 

goodtech@goodtech-desktop:~$ 
goodtech@goodtech-desktop:~$ sudo nano /etc/systemd/system/sshd.service
[sudo] password for goodtech: 
goodtech@goodtech-desktop:~$ 
goodtech@goodtech-desktop:~$ 
goodtech@goodtech-desktop:~$ sudo systemctl daemon-reload
goodtech@goodtech-desktop:~$ sudo systemctl start sshd
goodtech@goodtech-desktop:~$ sudo systemctl status sshd
● sshd.service - OpenSSH server daemon
     Loaded: loaded (/etc/systemd/system/sshd.service; disabled; preset: enable>
     Active: activating (auto-restart) (Result: exit-code) since Sat 2024-07-20>
    Process: 7897 ExecStart=/usr/local/sbin/sshd -D (code=exited, status=203/EX>
   Main PID: 7897 (code=exited, status=203/EXEC)
        CPU: 1ms

Jul 20 17:41:53 goodtech-desktop systemd[1]: sshd.service: Failed with result '>
goodtech@goodtech-desktop:~$ 
goodtech@goodtech-desktop:~$ 
goodtech@goodtech-desktop:~$ sudo systemctl enable sshd
Created symlink /etc/systemd/system/multi-user.target.wants/sshd.service → /etc/systemd/system/sshd.service.
goodtech@goodtech-desktop:~$ sudo systemctl status sshd
● sshd.service - OpenSSH server daemon
     Loaded: loaded (/etc/systemd/system/sshd.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Sat 2024-07-20>
   Main PID: 7897 (code=exited, status=203/EXEC)
        CPU: 1ms

Jul 20 17:41:53 goodtech-desktop systemd[1]: sshd.service: Failed with result '>
goodtech@goodtech-desktop:~$

 

 

 

 

goodtech@goodtech-desktop:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.67  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::2ecf:67ff:fe2a:726d  prefixlen 64  scopeid 0x20<link>
        ether 2c:cf:67:2a:72:6d  txqueuelen 1000  (Ethernet)
        RX packets 196250  bytes 276143385 (276.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 86168  bytes 7879149 (7.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 111  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3635  bytes 365079 (365.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3635  bytes 365079 (365.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 2c:cf:67:2a:72:6e  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

goodtech@goodtech-desktop:~$

 

 

 

 

 

 

Putty를 사용하여 접속한다.

 

goodtech@192.168.0.67's password:
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-1004-raspi aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

goodtech@goodtech-desktop:~$

 

 

 

고정 아이피 참고 우분투 24.04 에서

 

https://ko.linux-console.net/?p=31063

명령줄을 통해 Ubuntu 24.04에서 고정 IP 주소 설정

 

 

이게 파일이다. sudo nano /etc/netplan/50-cloud-init.yaml  

 

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eth0:
            dhcp4: no
            dhcp6: no
            addresses: [192.168.0.51/24]
            routes:
              - to: default
                via: 192.168.0.1
            nameservers:
                addresses: [8.8.8.8, 8.8.4.4]
    version: 2

 

 

 

자바 설치 참고 

 

https://velog.io/@ung6860/%EA%B0%9C%EB%B0%9C%ED%99%98%EA%B2%BDUbuntu%EC%97%90-Java-%EC%84%A4%EC%B9%98%ED%95%98%EA%B8%B0

 

 

 

호스트 이름 변경 pi050~pi053

 

호스트 변경

 

원래 데이터

 

127.0.0.1       localhost
127.0.1.1       goodtech-desktop

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

 

 

이걸 아래로 변경

 

# The following lines are desirable for IPv6 capable hosts
#::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

192.168.0.50 pi050
192.168.0.51 pi051
192.168.0.52 pi052
192.168.0.53 pi053

 

 

.ssh 폴더가 없어서

 

https://stackoverflow.com/questions/15190391/ssh-directory-not-being-created

.ssh directory not being created

 

107

I am assuming that you have enough permissions to create this directory.

To fix your problem, you can either ssh to some other location:

ssh user@some.host
and accept new key - it will create directory ~/.ssh and known_hosts underneath, or simply create it manually using

mkdir ~/.ssh
chmod 700 ~/.ssh
Note that chmod 700 is an important step!

After that, ssh-keygen should work without complaints.

 

 

 

 

공개 키 생성

 

goodtech@pi050:~$
goodtech@pi050:~$ ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/goodtech/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/goodtech/.ssh/id_rsa
Your public key has been saved in /home/goodtech/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:R4s4ynql18PA5+7okHReZ+Z0OtBneE5ZCa1Fev/Lj54 goodtech@pi050
The key's randomart image is:
+---[RSA 4096]----+
|            .o.  |
|             oo. |
|          . .o+  |
|       . + o.+ . |
|    ..+ S X B   .|
|   o =+o.O O    .|
|    =o.*  + .   .|
|   .o...=  .  .o.|
|  .. oooo.   .Eoo|
+----[SHA256]-----+
goodtech@pi050:~$
goodtech@pi050:~$

 

 

 

 

이 과정 수행 전체

 

goodtech@pi050:~$ ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/goodtech/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/goodtech/.ssh/id_rsa
Your public key has been saved in /home/goodtech/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:R4s4ynql18PA5+7okHReZ+Z0OtBneE5ZCa1Fev/Lj54 goodtech@pi050
The key's randomart image is:
+---[RSA 4096]----+
|            .o.  |
|             oo. |
|          . .o+  |
|       . + o.+ . |
|    ..+ S X B   .|
|   o =+o.O O    .|
|    =o.*  + .   .|
|   .o...=  .  .o.|
|  .. oooo.   .Eoo|
+----[SHA256]-----+
goodtech@pi050:~$
goodtech@pi050:~$ cat .ssh/id_rsa.pub  >> .ssh/authorized_keys
goodtech@pi050:~$
goodtech@pi050:~$
goodtech@pi050:~$ cat ~/.ssh/id_rsa.pub | ssh pi051 'cat >> .ssh/authorized_keys'
The authenticity of host '192.168.0.51 (192.168.0.51)' can't be established.
ED25519 key fingerprint is SHA256:sfpUMSaWlRzbC0jQbtsrWIvRckuWhh1NwVexWrycK0M.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.51' (ED25519) to the list of known hosts.
goodtec@192.168.0.51's password:
Permission denied, please try again.
goodtec@192.168.0.51's password:
Permission denied, please try again.
goodtec@192.168.0.51's password:
goodtec@192.168.0.51: Permission denied (publickey,password).
goodtech@pi050:~$ cat ~/.ssh/id_rsa.pub | ssh pi051 'cat >> .ssh/authorized_keys'
goodtec@192.168.0.51's password:
Permission denied, please try again.
goodtec@192.168.0.51's password:
Permission denied, please try again.
goodtec@192.168.0.51's password:
goodtec@192.168.0.51: Permission denied (publickey,password).
goodtech@pi050:~$ cat ~/.ssh/id_rsa.pub | ssh goodtech@pi051 'cat >> .ssh/authorized_keys'
goodtech@192.168.0.51's password:
goodtech@pi050:~$ cat ~/.ssh/id_rsa.pub | ssh goodtech@pi052 'cat >> .ssh/authorized_keys'
The authenticity of host '192.168.0.52 (192.168.0.52)' can't be established.
ED25519 key fingerprint is SHA256:NI+LA26uUZbZ6SiviKC/DQZMPTHqJ7f+xQ7Herak+X4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.52' (ED25519) to the list of known hosts.
goodtech@192.168.0.52's password:
goodtech@pi050:~$
goodtech@pi050:~$
goodtech@pi050:~$
goodtech@pi050:~$ cat ~/.ssh/id_rsa.pub | ssh goodtech@pi053 'cat >> .ssh/authorized_keys'
The authenticity of host '192.168.0.53 (192.168.0.53)' can't be established.
ED25519 key fingerprint is SHA256:Pe2IBdMAEc8M1dc1rxvK1o3SE2dEFdeTYVi9dNE6bPA.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.53' (ED25519) to the list of known hosts.
goodtech@192.168.0.53's password:
goodtech@pi050:~$
goodtech@pi050:~$

 

 

 

다 하고 나면 비번 없이 접속가능

 

goodtech@pi050:~$
goodtech@pi050:~$ ssh goodtech@pi051
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-1007-raspi aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status

Last login: Sat Jul 20 22:32:35 2024 from 192.168.0.2
goodtech@pi051:~$ exit
logout
Connection to 192.168.0.51 closed.
goodtech@pi050:~$

 

 

참고

라즈베리파이 yarn 클러스터에 하둡(hadoop)과 스파크(spark) 설치하기